Proposal: splitting APR and former apr-util libraries

[Graham Leggett <mi...@sharp.fm>]

Hi all,

A while back, the apr and apr-util projects were combined into a  
single project, however this amplified a problem we have been  
suffering for a while.

APR attempts to provide portable access to the following:

- Low level APIs, such as memory management, file access, process/ 
thread handing, etc, relying on basic OS functionality provided by  
glibc and friends.

- High level APIs, such as XML parsing, dbm and friends, SQL database  
and LDAP drivers, and crypto, relying on specific libraries providing  
each service.

The problem with mixing the two together is that many applications  
only care about the low level stuff, they don't want to link to  
BerkleyDB or expat just to get access to APR pools. This problem was  
in turn made worse when apr and apr-util was combined. It made sense  
that basic APIs like base64, date handling, string handling, etc  
should have moved to APR from apr-util, but what didn't make sense was  
moving everything.

Some APIs, such as the dbd SQL library, the LDAP library and the  
crypto API can be separated out from APR at build time on some  
platforms, and this is done by vendors like Redhat, who ship a range  
of APR packages containing the parts you need. This doesn't go far  
enough.

What I propose is that we take the APIs in the second group, including  
but not limited to XML, dbm, dbd, ldap and crypto, and spin these out  
into specific dedicated projects of their own under the APR umbrella,  
with their own release cycles, tags and branches. This would solve the  
following problems:

- APR proper would be the Apache Portable Runtime, offering low level  
API compatibility, and nothing more.

- Users of APR proper would never need to worry about transitively  
depending on libraries other than the core system libraries, such as  
expat.

- People wanting to add new types of portability APIs can do so  
without code bloat in APR proper being a problem.

- If apr-foo becomes abandoned, it can be sent to the attic in the  
normal way. This will solve wrowe's issues with the LDAP APIs, using  
established ASF practice.

Regards,
Graham
--

Re: Proposal: splitting APR and former apr-util libraries

[Joe Orton <jo...@redhat.com>]

On Wed, Jun 01, 2011 at 07:27:20PM -0400, Greg Stein wrote:
> You're just talking about bringing back apr-util.

The motivation for combining the apr and apr-util "projects" (separate 
tarball distributions) was that there we never had enough "churn" in 
either to justify the additional overhead of a separate RM cycle for 
each.

So I agree we don't need to go back to even higher RM overhead with lots 
of "projects".  But I agree totally with Graham that splitting out the 
sub-components into separate *libraries* is good and useful, so as to 
isolate deps on third-party libraries; that's always been the plan 
AFAIK.  I did at least start the groundwork on the buildsystem support 
for that (r758206).

Regards, Joe

Re: Proposal: splitting APR and former apr-util libraries

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 6/3/2011 11:18 AM, Justin Erenkrantz wrote:
> On Wed, Jun 1, 2011 at 4:27 PM, Greg Stein <gs...@gmail.com> wrote:
>> Making these N different packages doesn't do much beyond just what the
>> old apr-util did. I guess it solves a bit of dependency stuff.
> 
> Having had to get a Windows build env up due to the WSAPoll()
> brokenness, I have to say that the multiple libraries is just a giant
> PITA.  Good riddance to apr-util and apr-iconv...  =)  -- justin

There is still the complexity of dependencies, of course.

To make it such that you don't necessarily have to integrate all of the
possible dependencies, I intend to play some config games using .vbs,
which is available now always with the windows scripting host on all
shipping win32 OSs.

And if you simply rebuild the core and use an existing set of apr-2/*.dll
stubs, that should prove OK, too.  Will have to give that part some thought.

Re: Proposal: splitting APR and former apr-util libraries

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Wed, Jun 1, 2011 at 4:27 PM, Greg Stein <gs...@gmail.com> wrote:
> Making these N different packages doesn't do much beyond just what the
> old apr-util did. I guess it solves a bit of dependency stuff.

Having had to get a Windows build env up due to the WSAPoll()
brokenness, I have to say that the multiple libraries is just a giant
PITA.  Good riddance to apr-util and apr-iconv...  =)  -- justin

Re: Proposal: splitting APR and former apr-util libraries

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 6/3/2011 12:25 AM, William A. Rowe Jr. wrote:
> On 6/3/2011 12:05 AM, Greg Stein wrote:
>> On Thu, Jun 2, 2011 at 07:51, Graham Leggett <mi...@sharp.fm> wrote:
>>
>>> It seems what we're working towards is combining apr and apr-util, removing
>>> most of the stuff that was in apr-util, ending up pretty much back with apr,
>>> which leads me to ask why we ever bothered combining the two in the first
>>> place.
>>
>> Dunno. Go do the research in the svn logs. Somebody did the commit.
>> Look for the discussion around then, or ask that committer.
> 
> None of the merging of apr and apr-util would have even been
> considered, until we had factored out all of the library dependencies
> into sub-components of apr which *need not be loaded*.  Even apr 1.x
> core itself had too many dependencies which weren't necessary for
> each and every consumer.

** Also, if a library doesn't benefit from abstraction using pool
memory mechanics, it isn't likely to be a good candidate either.

Re: Proposal: splitting APR and former apr-util libraries

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 6/3/2011 12:05 AM, Greg Stein wrote:
> On Thu, Jun 2, 2011 at 07:51, Graham Leggett <mi...@sharp.fm> wrote:
> 
>> It seems what we're working towards is combining apr and apr-util, removing
>> most of the stuff that was in apr-util, ending up pretty much back with apr,
>> which leads me to ask why we ever bothered combining the two in the first
>> place.
> 
> Dunno. Go do the research in the svn logs. Somebody did the commit.
> Look for the discussion around then, or ask that committer.

None of the merging of apr and apr-util would have even been
considered, until we had factored out all of the library dependencies
into sub-components of apr which *need not be loaded*.  Even apr 1.x
core itself had too many dependencies which weren't necessary for
each and every consumer.

So dbd, dbm, even ldap were no longer triggered by 'linking to apr'.
Totally agreed with Greg in that older discussion that this was wrong.

Of course, ldap triggers the need to rebind the consumer application
to the same ldap library as apr was built against, so it violated all
of the separation principals.

But before we ship 2.0, it will be good to also lose the dependency
on always loading iconv, and expat/libxml2, and any other remnants
of being both a generalization library for the os/kernel, and also
a generalization library for most commonly required function libraries.

Re: Proposal: splitting APR and former apr-util libraries

[Greg Stein <gs...@gmail.com>]

On Thu, Jun 2, 2011 at 07:51, Graham Leggett <mi...@sharp.fm> wrote:
>...
> If people are starting their own portability libraries, then it shows that
> apr is not fit for purpose in its current form, and that needs to be
> addressed by the apr project. I don't recall much discussion happening over
> "combined vs split", suddenly we were combined, and as I recall nobody
> provided an explanation as to what problem they were trying to solve by
> doing so.

I tried to bring up discussion about concentrating on just OS
portability and dropping all the other gunk a couple years ago. See
the thread titled "APR: Portable across Operating Systems, or
Libraries?" back in January 2009. It kinda devolved into just talking
about platforms.

The part about combining... I don't know where exactly that came from.
I remember some discussion, but I was already thinking the
functionality should be dropped (whether merged or not).

> It seems what we're working towards is combining apr and apr-util, removing
> most of the stuff that was in apr-util, ending up pretty much back with apr,
> which leads me to ask why we ever bothered combining the two in the first
> place.

Dunno. Go do the research in the svn logs. Somebody did the commit.
Look for the discussion around then, or ask that committer.

I've never supported it, so I just stopped worrying about it since the
community seemed supportive of that direction... *shrug*

Cheers,
-g

Re: Proposal: splitting APR and former apr-util libraries

[Graham Leggett <mi...@sharp.fm>]

On 02 Jun 2011, at 1:27 AM, Greg Stein wrote:

> You're just talking about bringing back apr-util.

Pretty much, without the "lump it all together in apr-util".

> Making these N different packages doesn't do much beyond just what the
> old apr-util did. I guess it solves a bit of dependency stuff.

I disagree, it goes a lot further - it solves the problem of what to  
do if an API is proposed for APR that is simply too big for the library.

At this point we have no "escape valve" to protect us against code  
bloat. Our only option at the moment is to keep adding stuff to the  
core apr project, or just rejecting code with the explanation "we're  
full, go away". Neither of these are healthy options.

> Personally, I gave up on the "combined vs split" argument a long time
> ago. This community wanted to combine everything, and I felt
> differently. It happens. I also felt that APR has got a lot of old,
> clunky code that needs to be cleaned up. As a result, I started my
> PoCore project to deal with just the low-level portability concerns,
> with a full revisit of all the concepts based on experience with APR,
> HTTPD, svn, and serf. High-level stuff can be left to applications or
> other third-party libraries.

If people are starting their own portability libraries, then it shows  
that apr is not fit for purpose in its current form, and that needs to  
be addressed by the apr project. I don't recall much discussion  
happening over "combined vs split", suddenly we were combined, and as  
I recall nobody provided an explanation as to what problem they were  
trying to solve by doing so.

It seems what we're working towards is combining apr and apr-util,  
removing most of the stuff that was in apr-util, ending up pretty much  
back with apr, which leads me to ask why we ever bothered combining  
the two in the first place.

Regards,
Graham
--

Re: Proposal: splitting APR and former apr-util libraries

[Greg Stein <gs...@gmail.com>]

On Wed, Jun 1, 2011 at 18:37, Graham Leggett <mi...@sharp.fm> wrote:
> Hi all,
>
> A while back, the apr and apr-util projects were combined into a single
> project, however this amplified a problem we have been suffering for a
> while.
>
> APR attempts to provide portable access to the following:
>
> - Low level APIs, such as memory management, file access, process/thread
> handing, etc, relying on basic OS functionality provided by glibc and
> friends.

This is what "apr" used to be.

>
> - High level APIs, such as XML parsing, dbm and friends, SQL database and
> LDAP drivers, and crypto, relying on specific libraries providing each
> service.

And this is what "apr-util" used to be.

>
> The problem with mixing the two together is that many applications only care
> about the low level stuff, they don't want to link to BerkleyDB or expat
> just to get access to APR pools. This problem was in turn made worse when
> apr and apr-util was combined. It made sense that basic APIs like base64,
> date handling, string handling, etc should have moved to APR from apr-util,
> but what didn't make sense was moving everything.
>
> Some APIs, such as the dbd SQL library, the LDAP library and the crypto API
> can be separated out from APR at build time on some platforms, and this is
> done by vendors like Redhat, who ship a range of APR packages containing the
> parts you need. This doesn't go far enough.
>
> What I propose is that we take the APIs in the second group, including but
> not limited to XML, dbm, dbd, ldap and crypto, and spin these out into
> specific dedicated projects of their own under the APR umbrella, with their
> own release cycles, tags and branches. This would solve the following
> problems:

You're just talking about bringing back apr-util.

Making these N different packages doesn't do much beyond just what the
old apr-util did. I guess it solves a bit of dependency stuff.

>...

Personally, I gave up on the "combined vs split" argument a long time
ago. This community wanted to combine everything, and I felt
differently. It happens. I also felt that APR has got a lot of old,
clunky code that needs to be cleaned up. As a result, I started my
PoCore project to deal with just the low-level portability concerns,
with a full revisit of all the concepts based on experience with APR,
HTTPD, svn, and serf. High-level stuff can be left to applications or
other third-party libraries.

Cheers,
-g

Re: Proposal: splitting APR and former apr-util libraries

["William A. Rowe Jr." <wr...@rowe-clan.net>]

On 6/1/2011 5:37 PM, Graham Leggett wrote:
> 
> What I propose is that we take the APIs in the second group, including but not limited to
> XML, dbm, dbd, ldap and crypto, and spin these out into specific dedicated projects of
> their own under the APR umbrella, with their own release cycles, tags and branches. This
> would solve the following problems:

[Include xlate (iconv) in that list]

-1, and I give you apr-iconv and apr-ldap both as the proof that this
is a doomed proposal, and dbd as counterexample of why this is not the
desirable state in the first place.  Actually, I'd offer the pain of the
whole state of PECL as an even better example, and CPAN as an example of
what is necessary to make it work.

dbd was stand alone long before it became part of apr-util and I would be
very interested in niq's opinion.