Opaque types [was apr_mmap_t]

["William A. Rowe, Jr." <wr...@apache.org>]

At 10:07 AM 11/24/2002, rbb@apache.org wrote:

>Let me make myself clear.  I haven't expressed an opinion about whether
>they should or should not be incomplete.  I have stated why incomplete
>types were originally used in APR, namely for portability.  

Some of us interpreted (rightly or wrongly) that they benefit us in a number
of ways.  There is more than one [right] opinion on this forum.

>I have also said that having MMAPs be represented by anything other 
>than a void pointer and a size is a bit ludicrous.

Then you clearly don't understand the implementation.

Win32 must retrieve page-aligned file blocks.  That means that the real
pointer to the mmap allocation is not the boundry the user requested,
because we backspace to the last boundry when mapping the region.
Then we give the user a pointer to their desired offset.

Worse yet, the mmap is not the base kernel object.  All mmap's start
their lives with a handle to a mapping.  With that handle to the mapping,
we then map any desired pages into memory.

So your offset/length assertion is, simply put, somewhat bogus.  We
cannot even release the allocation given that original offset/length, on
Win32 we must release the page-aligned region, then the mapping handle.
I'll skip your other comments as they aren't germane to a portable mmap
implementation.

>I thought of this a few months ago, but I have been sitting on it while we
>try to get 1.0 out the door.  This is my vision for how APR 2.0 will be
>implemented (I was going to propose it to the list after 1.0 had been
>released).  This makes the API cleaner and smaller, because many of the
>accessor functions can cleanly be deprecated (Think all of the userdata
>functions).  It also solves a lot of problems with apr_proc_t's on
>Windows, where we don't have anyplace to store OS specific information.
>
>When I said a few days ago that I had hoped that many of the incomplete
>types would go away in APR 2.0, this is what I meant.  I don't think we
>should re-write APR 1.0 now, but if you are going to change MMAPs, then
>please let's do it the right way.

Your "one right way" seems to be at odds with other's interpretations.

I'm not arguing that exposing incomplete types wouldn't speed up APR.
In fact, well written accessors that are inline'able on all modern compilers
would be a beautiful thing.  They would work only for 'private' copies of
the apr library, since the app becomes permanently bound to the explicit
version installed at the time you build it.  That would be fine by httpd and
other consumers who don't mind building APR themselves, or for apps
that are built via one of the packaging tools that tracks dependencies,
and will rebuild the entire slew of dependent packages when a new 
version of APR is installed.

Of course, this won't work for deployment across different platforms.
By committing to unwrapping all of the opaque types, you seem hell
bent on ensuring we forever break binary compatibility.  Opaque types
are goodness for versioning, portability, and extensions of the APR
library to other languages (e.g. tcl, perl, java etc, especially those
with OO schemas).  We (you especially) did a passable job of making
APR map very well to OO schemas.  Please consider their benefits 
of opaque types before we make plans to wipe them out.

Bill

Re: Opaque types [was apr_mmap_t]

[rb...@apache.org]

On Mon, 25 Nov 2002, William A. Rowe, Jr. wrote:

> At 11:51 PM 11/24/2002, rbb@apache.org wrote:
> >On Sun, 24 Nov 2002, William A. Rowe, Jr. wrote:
> >>
> >> Obviously, this is skeletal.  It provides that all functions are exported,
> >> and some simple functions may be inlined.
> >
> >I tried to snip some of this, but I couldn't.  This is bogus.  Your header
> >with the inline header needs information that is private to the APR
> >internals.  That means that it can never be included from any file that is
> >public.  At the end of the day, what you are describing above is
> >completely impossible.
>
> Obviously it marries the app to the explicit version of APR.  That is the
> downside of inlining otherwise innocuous operations.

No, it does much more than that.  It completely removes the incompleteness
of the type.  Think it through.  The header that has the inline function
(the private header) needs to have full access to the internals of the
type.  The public header CAN'T have access to the internals of the type.
In order for the inlining to work, the private header needs to be included
from an external file (or the public header, which is the same thing).
This means that once you track it all back, the app has access to the
definition of the incomplete type, making it no longer incomplete.

If you can somehow make this work without removing the incompleteness of
APR, then cool.  But, AFAICS, it isn't possible, because what you want to
do can't be done with incomplete types.

> >> I don't believe what you suggest is portable.  Of course my VC is very happy
> >> to parse a struct def with a pointer to an incomplete type anywhere within
> >> the structure, or an undefined array at the *end* of the structure.  But since
> >
> >Will, please, that is just completely bogus.  Are you honestly telling me
> >that you believe that there is a compiler that doesn't know how to put a
> >pointer to an incomplete type inside of a structure?  If so, then Apache
>
> No, I'm stating that having a partially complete structure is not portable.
>
> The question of an embedded pointer to the apr_foo_internal_t isn't an
> issue, except that the user can allocate an apr_foo_t.  Since I just got
> done earlier today expressing that we don't care that users can shoot
> themselves in the foot, I really don't have any argument to that side
> of your proposal.

So in other words, the idea that I have expressed is completely portable.
Cool, then what was your problem with it?????

> >2.0 isn't portable to that platform.  What I am suggesting is 100%
> >identical to what Apache does by putting an apr_file_t inside the
> >request_rec.  Add to that, we are ALREADY doing what I suggest, this model
> >was used in apr_poll_t, and it works just find.
>
> And it implies that apr_foo_t could be allocated by the user, although they
> cannot create the pointer to apr_foo_internal_t.  Let me digest what you are
> proposing so I can comment on it further.

Yep, and without the pointer to the internal structure, APR is essentially
useless.  So, while it would be possible to allocate the public portion of
the structure, it wouldn't get the user anywhere.  Which means, that
binary compat remains, because APR remains the only valid way to allocate
an APR structure.

The only place that binary compat becomes a problem, is that people can
now pass the structure instead of a pointer to the structure.  But if
somebody wants to shoot themselves in the foot, then go ahead.  All they
will be losing is binary compat, which isn't the end of the world,
especially since we provide an easy way for them to regain it.

> You are suggesting two structures.  I spelled out in the prior message
> and a paragraph above why that was a half-way solution.  The real
> answer is a partially complete structure; which is not ANSI C or any
> flavor thereof.

I obviously missed your explanation of why that is a half-way solution.
I'll try to go back and read it again to find the reasoning, but it may
take a day or two.

Ryan

Re: Opaque types [was apr_mmap_t]

["William A. Rowe, Jr." <wr...@apache.org>]

At 11:51 PM 11/24/2002, rbb@apache.org wrote:
>On Sun, 24 Nov 2002, William A. Rowe, Jr. wrote:
>>
>> Obviously, this is skeletal.  It provides that all functions are exported,
>> and some simple functions may be inlined.
>
>I tried to snip some of this, but I couldn't.  This is bogus.  Your header
>with the inline header needs information that is private to the APR
>internals.  That means that it can never be included from any file that is
>public.  At the end of the day, what you are describing above is
>completely impossible.

Obviously it marries the app to the explicit version of APR.  That is the
downside of inlining otherwise innocuous operations.

>> I don't believe what you suggest is portable.  Of course my VC is very happy
>> to parse a struct def with a pointer to an incomplete type anywhere within
>> the structure, or an undefined array at the *end* of the structure.  But since
>
>Will, please, that is just completely bogus.  Are you honestly telling me
>that you believe that there is a compiler that doesn't know how to put a
>pointer to an incomplete type inside of a structure?  If so, then Apache

No, I'm stating that having a partially complete structure is not portable.

The question of an embedded pointer to the apr_foo_internal_t isn't an
issue, except that the user can allocate an apr_foo_t.  Since I just got
done earlier today expressing that we don't care that users can shoot
themselves in the foot, I really don't have any argument to that side
of your proposal.

>2.0 isn't portable to that platform.  What I am suggesting is 100%
>identical to what Apache does by putting an apr_file_t inside the
>request_rec.  Add to that, we are ALREADY doing what I suggest, this model
>was used in apr_poll_t, and it works just find.

And it implies that apr_foo_t could be allocated by the user, although they
cannot create the pointer to apr_foo_internal_t.  Let me digest what you are
proposing so I can comment on it further.

>I am not going to respond to the rest of this message, it is nothing but
>flame bait.

Glad we are having a discussion and not a debate, then.

You are suggesting two structures.  I spelled out in the prior message
and a paragraph above why that was a half-way solution.  The real
answer is a partially complete structure; which is not ANSI C or any
flavor thereof.

>We are still waiting for the real labor to begin.  I has been two days,
>and I am wasting time on the computer.

I hear that ... no rest for the expectant mother -or- father :-) 

Bill

Re: Opaque types [was apr_mmap_t]

[rb...@apache.org]

On Sun, 24 Nov 2002, William A. Rowe, Jr. wrote:

> At 09:24 PM 11/24/2002, rbb@apache.org wrote:
>
> >> I'm not arguing that exposing incomplete types wouldn't speed up APR.
> >> In fact, well written accessors that are inline'able on all modern compilers
> >> would be a beautiful thing.  They would work only for 'private' copies of
> >> the apr library, since the app becomes permanently bound to the explicit
> >> version installed at the time you build it.  That would be fine by httpd and
> >
> >There are very few compilers that can inline functions that are in
> >separate libraries.  In fact, I don't currently know of any.
>
> Of course, there are none that inline functions in libraries.  inline functions
> need to exist in headers.  We might end up with a scenario such as...
>
> apr_mmap.h  [public header]
>   #include "apr.h"
>
>   /* Incomplete type */
>   typedef struct apr_mmap_t apr_mmap_t;
>
>   /* Prototype */
>   APR_DECLARE_INLINE(footype) apr_mmap_foo_get(apr_mmap_t *mmap);
>
>   #if APR_HAS_INLINE && APR_USE_INLINE
>   # include "apr_mmap_inline.h"
>   #endif
>
> apr_mmap_inline.h
>   /* This private header is included for inline declarations.
>    * use of the embedded structures binds your application
>    * to this specific build of APR.  These structures are subject
>    * to change without warning. */
>
>   struct apr_mmap_t {
>     footype foo; ...
>   } apr_mmap_t;
>
>   /* Implementation */
>   APR_DECLARE_INLINE(typefoo) apr_mmap_foo_get(apr_mmap_t *mmap) {
>
>   }
>
> mmap_inline.c
>   #undef APR_USE_INLINE
>   #include "apr_mmap.h"
>
>   /* capture inline functions as exported real functions */
>   #include "apr_mmap_inline.h"
>
> mmap.c
>   #define APR_USE_INLINE  /* if it's not simply global */
>   #include "apr_mmap.h"
>
>   APR_DECLARE(apr_status_t) apr_mmap_create(...) {
>     ...
>   }
>
> Obviously, this is skeletal.  It provides that all functions are exported,
> and some simple functions may be inlined.

I tried to snip some of this, but I couldn't.  This is bogus.  Your header
with the inline header needs information that is private to the APR
internals.  That means that it can never be included from any file that is
public.  At the end of the day, what you are describing above is
completely impossible.

> >> other consumers who don't mind building APR themselves, or for apps
> >> that are built via one of the packaging tools that tracks dependencies,
> >> and will rebuild the entire slew of dependent packages when a new
> >> version of APR is installed.
> >>
> >> Of course, this won't work for deployment across different platforms.
> >
> >I can't understand this statement at all.  Deployment across different
> >platforms doesn't make sense in APR.
>
> Sorry, wrong phrase.  Different target machines.  Think binary builds of
> packages, where the package doesn't distribute APR, but presumes it
> will be installed on the machine.  Binary compatibility to the inlines would
> never fly, so that package builds without APR_USE_INLINE, for example.
> Anyone building the package to include the specific release of APR would
> use inline, of course.

As I said above, your whole inlining concept won't work.  You are trying
to get private data to public headers, which can't happen.

> >> By committing to unwrapping all of the opaque types, you seem hell
> >> bent on ensuring we forever break binary compatibility.  Opaque types
> >
> >Again, what in the world are you talking about?  The design that I
> >suggested actually makes binary compatibility one of it's core goals.  I
> >am a VERY strong believer in binary compat for libraries, so this
> >statement makes little to no sense.  If you look at the design, you will
> >see that it splits the structure into two parts:  1)  The logical
> >component and 2)  The OS specific component.  Because there is an
> >incomplete type in the middle of the structure, an allocator is still
> >required to actually create an instance of a type.  That means that binary
> >compat is easy to get, assuming all new fields are added at the end of the
> >structure.  The only time you will get into trouble is if somebody passes
> >the structure itself instead of a pointer to the structure.  Yes, binary
> >compat would be broken between 1.0 and 2.0, but we never said that was a
> >goal.
>
> I don't believe what you suggest is portable.  Of course my VC is very happy
> to parse a struct def with a pointer to an incomplete type anywhere within
> the structure, or an undefined array at the *end* of the structure.  But since

Will, please, that is just completely bogus.  Are you honestly telling me
that you believe that there is a compiler that doesn't know how to put a
pointer to an incomplete type inside of a structure?  If so, then Apache
2.0 isn't portable to that platform.  What I am suggesting is 100%
identical to what Apache does by putting an apr_file_t inside the
request_rec.  Add to that, we are ALREADY doing what I suggest, this model
was used in apr_poll_t, and it works just find.

I am not going to respond to the rest of this message, it is nothing but
flame bait.

> Let me close by offering that we *better* not find out you are responding
> to list messages while waiting in the delivery room/suite :-)  We can pick
> up the dialog after you get back.  God bless the three of you :-)

We are still waiting for the real labor to begin.  I has been two days,
and I am wasting time on the computer.

Ryan

Re: Opaque types [was apr_mmap_t]

["William A. Rowe, Jr." <wr...@apache.org>]

At 09:24 PM 11/24/2002, rbb@apache.org wrote:

>> I'm not arguing that exposing incomplete types wouldn't speed up APR.
>> In fact, well written accessors that are inline'able on all modern compilers
>> would be a beautiful thing.  They would work only for 'private' copies of
>> the apr library, since the app becomes permanently bound to the explicit
>> version installed at the time you build it.  That would be fine by httpd and
>
>There are very few compilers that can inline functions that are in
>separate libraries.  In fact, I don't currently know of any.

Of course, there are none that inline functions in libraries.  inline functions
need to exist in headers.  We might end up with a scenario such as...

apr_mmap.h  [public header]
  #include "apr.h"

  /* Incomplete type */
  typedef struct apr_mmap_t apr_mmap_t;

  /* Prototype */
  APR_DECLARE_INLINE(footype) apr_mmap_foo_get(apr_mmap_t *mmap);

  #if APR_HAS_INLINE && APR_USE_INLINE
  # include "apr_mmap_inline.h"
  #endif

apr_mmap_inline.h 
  /* This private header is included for inline declarations.
   * use of the embedded structures binds your application
   * to this specific build of APR.  These structures are subject
   * to change without warning. */

  struct apr_mmap_t {
    footype foo; ...
  } apr_mmap_t;

  /* Implementation */
  APR_DECLARE_INLINE(typefoo) apr_mmap_foo_get(apr_mmap_t *mmap) {
    
  }

mmap_inline.c
  #undef APR_USE_INLINE
  #include "apr_mmap.h"

  /* capture inline functions as exported real functions */
  #include "apr_mmap_inline.h"

mmap.c
  #define APR_USE_INLINE  /* if it's not simply global */
  #include "apr_mmap.h"

  APR_DECLARE(apr_status_t) apr_mmap_create(...) {
    ...
  }

Obviously, this is skeletal.  It provides that all functions are exported,
and some simple functions may be inlined.

>> other consumers who don't mind building APR themselves, or for apps
>> that are built via one of the packaging tools that tracks dependencies,
>> and will rebuild the entire slew of dependent packages when a new
>> version of APR is installed.
>>
>> Of course, this won't work for deployment across different platforms.
>
>I can't understand this statement at all.  Deployment across different
>platforms doesn't make sense in APR.

Sorry, wrong phrase.  Different target machines.  Think binary builds of
packages, where the package doesn't distribute APR, but presumes it
will be installed on the machine.  Binary compatibility to the inlines would
never fly, so that package builds without APR_USE_INLINE, for example.
Anyone building the package to include the specific release of APR would 
use inline, of course.

>> By committing to unwrapping all of the opaque types, you seem hell
>> bent on ensuring we forever break binary compatibility.  Opaque types
>
>Again, what in the world are you talking about?  The design that I
>suggested actually makes binary compatibility one of it's core goals.  I
>am a VERY strong believer in binary compat for libraries, so this
>statement makes little to no sense.  If you look at the design, you will
>see that it splits the structure into two parts:  1)  The logical
>component and 2)  The OS specific component.  Because there is an
>incomplete type in the middle of the structure, an allocator is still
>required to actually create an instance of a type.  That means that binary
>compat is easy to get, assuming all new fields are added at the end of the
>structure.  The only time you will get into trouble is if somebody passes
>the structure itself instead of a pointer to the structure.  Yes, binary
>compat would be broken between 1.0 and 2.0, but we never said that was a
>goal.

I don't believe what you suggest is portable.  Of course my VC is very happy
to parse a struct def with a pointer to an incomplete type anywhere within
the structure, or an undefined array at the *end* of the structure.  But since
we can't partially define a single structure, we will end up with either a pointer
and two allocs, or some cruft to perform a single allocation and stash a pointer
or 'presume' that the opaque component follows the transparent structure.
Either way, you still end up with a legitimate sizeof() unless the structure
allows some incomplete component at the end of the declaration.  Some
compilers don't. 

>> are goodness for versioning, portability, and extensions of the APR
>> library to other languages (e.g. tcl, perl, java etc, especially those
>> with OO schemas).  We (you especially) did a passable job of making
>> APR map very well to OO schemas.  Please consider their benefits
>> of opaque types before we make plans to wipe them out.
>
>Pay very close attention please.  Using a combination of complete and
>incomplete types allows for all of the advantages of both, with very few
>disadvantages.

Yes, but as I just stated in the last paragraph, I don't believe it's available
to all compilers.

>The idea of wrapping APR in an OO language has never been called out as a
>goal, but even if it is made into a goal, splitting the types into a
>combination of complete and incomplete will not hurt that goal at all.

No, it would not if that facility were portable :-)

Let me close by offering that we *better* not find out you are responding
to list messages while waiting in the delivery room/suite :-)  We can pick
up the dialog after you get back.  God bless the three of you :-)  

Bill

Re: Opaque types [was apr_mmap_t]

[rb...@apache.org]

On Sun, 24 Nov 2002, William A. Rowe, Jr. wrote:

> At 10:07 AM 11/24/2002, rbb@apache.org wrote:
>
> >Let me make myself clear.  I haven't expressed an opinion about whether
> >they should or should not be incomplete.  I have stated why incomplete
> >types were originally used in APR, namely for portability.
>
> Some of us interpreted (rightly or wrongly) that they benefit us in a number
> of ways.  There is more than one [right] opinion on this forum.

Understand something:  The original reason for using incomplete types
ISN'T an opinion.  It is a point of fact.  The reasons that we all like
them is an opinion, and I haven't said anything about that.

> >I have also said that having MMAPs be represented by anything other
> >than a void pointer and a size is a bit ludicrous.
>
> Then you clearly don't understand the implementation.
>
> Win32 must retrieve page-aligned file blocks.  That means that the real
> pointer to the mmap allocation is not the boundry the user requested,
> because we backspace to the last boundry when mapping the region.
> Then we give the user a pointer to their desired offset.
>
> Worse yet, the mmap is not the base kernel object.  All mmap's start
> their lives with a handle to a mapping.  With that handle to the mapping,
> we then map any desired pages into memory.
>
> So your offset/length assertion is, simply put, somewhat bogus.  We
> cannot even release the allocation given that original offset/length, on
> Win32 we must release the page-aligned region, then the mapping handle.
> I'll skip your other comments as they aren't germane to a portable mmap
> implementation.

No, it isn't bogus.  The simple fact is that to the application developer,
an MMAP is a void pointer and an offset.  Anything else that is in the
structure is baggage that the OS requires us to carry.  Shrink the MMAP
structure to what it REALLY means to the developer and hide everything
else.

< snipped the rest, I'll respond in a different message, to split MMAP
from future design>

Ryan

Re: Opaque types [was apr_mmap_t]

[rb...@apache.org]

> >I thought of this a few months ago, but I have been sitting on it while we
> >try to get 1.0 out the door.  This is my vision for how APR 2.0 will be
> >implemented (I was going to propose it to the list after 1.0 had been
> >released).  This makes the API cleaner and smaller, because many of the
> >accessor functions can cleanly be deprecated (Think all of the userdata
> >functions).  It also solves a lot of problems with apr_proc_t's on
> >Windows, where we don't have anyplace to store OS specific information.
> >
> >When I said a few days ago that I had hoped that many of the incomplete
> >types would go away in APR 2.0, this is what I meant.  I don't think we
> >should re-write APR 1.0 now, but if you are going to change MMAPs, then
> >please let's do it the right way.
>
> Your "one right way" seems to be at odds with other's interpretations.

Nobody has said that yet.  In fact, your response is the first one since I
posted the idea.  Regardless, the statement stands.  If you are going to
rewrite MMAPS, then please let's do it the right way.  Whether that way is
what I suggested or not is immaterial.

> I'm not arguing that exposing incomplete types wouldn't speed up APR.
> In fact, well written accessors that are inline'able on all modern compilers
> would be a beautiful thing.  They would work only for 'private' copies of
> the apr library, since the app becomes permanently bound to the explicit
> version installed at the time you build it.  That would be fine by httpd and

There are very few compilers that can inline functions that are in
separate libraries.  In fact, I don't currently know of any.

> other consumers who don't mind building APR themselves, or for apps
> that are built via one of the packaging tools that tracks dependencies,
> and will rebuild the entire slew of dependent packages when a new
> version of APR is installed.
>
> Of course, this won't work for deployment across different platforms.

I can't understand this statement at all.  Deployment across different
platforms doesn't make sense in APR.

> By committing to unwrapping all of the opaque types, you seem hell
> bent on ensuring we forever break binary compatibility.  Opaque types

Again, what in the world are you talking about?  The design that I
suggested actually makes binary compatibility one of it's core goals.  I
am a VERY strong believer in binary compat for libraries, so this
statement makes little to no sense.  If you look at the design, you will
see that it splits the structure into two parts:  1)  The logical
component and 2)  The OS specific component.  Because there is an
incomplete type in the middle of the structure, an allocator is still
required to actually create an instance of a type.  That means that binary
compat is easy to get, assuming all new fields are added at the end of the
structure.  The only time you will get into trouble is if somebody passes
the structure itself instead of a pointer to the structure.  Yes, binary
compat would be broken between 1.0 and 2.0, but we never said that was a
goal.

> are goodness for versioning, portability, and extensions of the APR
> library to other languages (e.g. tcl, perl, java etc, especially those
> with OO schemas).  We (you especially) did a passable job of making
> APR map very well to OO schemas.  Please consider their benefits
> of opaque types before we make plans to wipe them out.

Pay very close attention please.  Using a combination of complete and
incomplete types allows for all of the advantages of both, with very few
disadvantages.

The idea of wrapping APR in an OO language has never been called out as a
goal, but even if it is made into a goal, splitting the types into a
combination of complete and incomplete will not hurt that goal at all.

Ryan