You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by William A Rowe Jr <wr...@rowe-clan.net> on 2018/02/20 19:20:24 UTC
Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
On Mon, Feb 19, 2018 at 12:50 PM, Joe Orton <jo...@redhat.com> wrote:
> On Sat, Feb 17, 2018 at 02:06:20PM -0000, minfrin@apache.org wrote:
>> Author: minfrin
>> Date: Sat Feb 17 14:06:20 2018
>> New Revision: 1824592
>>
>> URL: http://svn.apache.org/viewvc?rev=1824592&view=rev
>> Log:
>> Update proposal with fix for rpluem/jorton.
>
> Extending dav_resource still breaks binary backards compat with all(?)
> consumers of this API, or am I missing something here?
>
> Look at what mod_dav_svn does with the struct for an example:
> https://svn.apache.org/viewvc/subversion/trunk/subversion/mod_dav_svn/repos.c?view=markup#l80
(We can continue this from top-thread, but was this question
posed to or already considered by dev@svn?)
> If breaking backwards compat for module API is fine for 2.4 (and I'm
> still generally on the fence about this), I think it should at least be
> a "hard" break, e.g. by s/dav_register_provider/dav_register_provider2/.
> That way incompatible module combinations break at LoadModule time
> rather than with segfaults.
This is our Announcement policy;
This release builds on and extends the Apache 2.2 API. Modules written
for Apache 2.2 will need to be recompiled in order to run with Apache
2.4, and require minimal or no source code changes.
http://svn.apache.org/repos/asf/httpd/httpd/trunk/VERSIONING
When upgrading or installing this version of Apache, please bear in mind
that if you intend to use Apache with one of the threaded MPMs (other
than the Prefork MPM), you must ensure that any modules you will be
using (and the libraries they depend on) are thread-safe.
Implication from the above, Modules written for Apache 2.4 will not need
to be recompiled in order to run with Apache 2.4.
Reading that linked document in reverse;
DEVELOPMENT RELEASES, 2.{odd}.{revision}
-----------------------------------------
All odd numbered releases designate the 'next' possible stable release,
therefore the current development version will always be one greater than
the current stable release. Work proceeds on development releases, permitting
the modification of the MMN at any time in order to correct deficiencies
or shortcomings in the API. This means that modules from one development
release to another may not be binary compatible, or may not successfully
compile without modification to accomodate the API changes.
In other words, modules from one STABLE release to another ARE binary
compatible and do NOT need to be recompiled.
This is clearly not true of several recent changes, even though they
impact relatively few third party modules.
I suggest that since the majority here accepted binary breakage as OK,
that the project at MINIMUM now documents every known third party
module that must be 1) modified in source or 2) recompiled for binary
breakage, and we place that full appendix in every Announcement, and
on the main httpd website announcement and download page (followed by
some "... and similar modules" caviet, since people may have forked
the logic from these third party examples.
Or I suggest we back out breaking changes and go back to what this
project previously agreed upon and promised for the lifespan of 2.4.x.
I don't believe we ever had unanimous consensus, the change to
ABI-strict versioning was a majority vote at the end of the day?
Changing that policy mid-version seems utterly stupid to me, but I'm
tired of people taking personal offense at -1's, so repeating what
I've said, I'm neither supporting nor opposing any 2.4.x changes.
Simply reminding the list of the basis for our policies.
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Tue, Feb 20, 2018 at 3:28 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> This probably does not apply to 2.4.x (as a strong statement), in the
> meantime we at least need the helpers and give a hand at updating the
> modules, if we can't avoid extending our own structs...
I agree this discussion, outside of an interesting balancer shm
change and proposed mod_dav enhancement change, should
be largely status-quo.
I will go so far as to veto the idea of backporting the helpers to 2.4.
We cannot avoid admitting our errors in design of that released
branch, and papering over them now accomplishes little. At best
it would require an mmn check and only optionally consider their
use at runtime, with no hard linkage, since such modules can't
be loaded in earlier 2.4.x httpd releases.
Decide our fixes for rev.next and make that available for authors
to have a much easier time keeping binary compatibility with us,
even in deep intrinsic structures as you hint at.
One way to do this is to rev specific structure definitions... if the
module wants to break over the way they customized their
interaction with the structure or its indirection to members
(e.g. const/non-const changes) then let them do that at the
structure-relevant revision.
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Feb 20, 2018 at 9:39 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> On Tue, Feb 20, 2018 at 2:24 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>
>> 1. We continue as we do now and allow extending structures at the end. We should probably document more prominently that
>> copying / allocating / creating public structures is not allowed and if done can require minor version specific recompiling.
>
[]
>
> The proposed mod_dav case is worse yet. If we accept that the
> API demands the submodule pass the struct to mod_dav, there
> is no way to distinguish what is going on, and whether this module
> was written for 2.4.27 or 2.4.33. Worse 2x yet, we guaranteed that
> no source code revision would be required during 2.4.x.
The issue is that there is nothing we can do for the mod_dav case
(besides not extending the struct at all).
Neither _sizeof nor _copy would address that, if a module struct
compounds an ap_xxx_rec something anywhere but at the end then bets
are off (not to talk about this struct being used as a persised file
record).
How to statically define such a struct when _sizeof() is a function?
Offsets make using less friendly to use at runtime too...
We possibly need to exclude compound structs from our ABI garantees.
>
>> 2. We stop allowing extending structures at the end on stable branches.
>
> I don't think that is realistic. Adding a new member with an acceptable
> NULL/unknown purpose, and adding 3) as noted below, should
> generally solve for the case of "module which must be recompiled
> on each release".
I hope we won't do 2., or we may need to release a major version quite
often (which wouldn't help module authors either).
>
>> 3. Like 1., but we add _sizeof / _copy functions for each public struct as discussed in the other thread, make the
>> existence of the functions mandatory for every new public struct and advice module authors that they need to leverage
>> them if the want to copy / allocate / create these structures in their modules.
>>
>> We can do 1., 2. or 3. now or set this as the rule for the next major release.
>
> The advantage to _sizeof/_copy and enforcing the "new member
> default must be ok-to-be-NULL" rule means that mod_cluster,
> mod_ftp, mod_wsgi and many many other edge-case modules
> don't need to be recompiled. This situation can be dramatically
> improved in 2.next/3.0. Ensuring _create() accessors are always
> provided and demanded to initialize everything that shouldn't
> default to NULL is the 100% solve.
I'm fine with 1 and 3, and would like that we don't guarantee ABI
stability for compound structs or persisted files with our structs as
record, such constructions should use 3. with runtime tricks (in the
first place, IMHO).
This probably does not apply to 2.4.x (as a strong statement), in the
meantime we at least need the helpers and give a hand at updating the
modules, if we can't avoid extending our own structs...
Regards,
Yann.
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Tue, Feb 20, 2018 at 3:43 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Tue, Feb 20, 2018 at 10:29 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>> On Tue, Feb 20, 2018 at 2:57 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>>
>>> On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
>>>
>>>> Moving a member in a well-defined structure doesn't fall into this
>>>> generally accepted change (expanding the length of a struct.)
>>>> Consider the shm array change doesn't fall into the "it is just
>>>> a longer struct" example.
>>>
>>> Can you please give me a pointer which shm array change you are referring to?
>>
>> r1824504
>
> I'm not sure it falls outside our guarantees, the old "hostname" field
> is still there (at the same place) and populated.
NUL \0 delimited? Cool.
Provided mod_status and mod_cluster 2.4.29 won't crash,
*for display purposes* we can agree that isn't a concern.
(mod_cluster updates will crash and we can agree that was
extending the API contract to an unrealistic bounds.)
Bill
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Feb 20, 2018 at 10:29 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> On Tue, Feb 20, 2018 at 2:57 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>
>> On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
>>
>>> Moving a member in a well-defined structure doesn't fall into this
>>> generally accepted change (expanding the length of a struct.)
>>> Consider the shm array change doesn't fall into the "it is just
>>> a longer struct" example.
>>
>> Can you please give me a pointer which shm array change you are referring to?
>
> r1824504
I'm not sure it falls outside our guarantees, the old "hostname" field
is still there (at the same place) and populated.
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Wed, Feb 21, 2018 at 12:50 AM, Plüm, Rüdiger, Vodafone Group
<ru...@vodafone.com> wrote:
>
>> -----Ursprüngliche Nachricht-----
>> Von: William A Rowe Jr [mailto:wrowe@rowe-clan.net]
>> Gesendet: Dienstag, 20. Februar 2018 22:29
>> An: httpd <de...@httpd.apache.org>
>> Betreff: Re: Binary Breakage (was: svn commit: r1824592 -
>> /httpd/httpd/branches/2.4.x/STATUS)
>>
>> On Tue, Feb 20, 2018 at 2:57 PM, Ruediger Pluem <rp...@apache.org>
>> wrote:
>> >
>> > On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
>> >
>> >> Moving a member in a well-defined structure doesn't fall into this
>> >> generally accepted change (expanding the length of a struct.)
>> >> Consider the shm array change doesn't fall into the "it is just
>> >> a longer struct" example.
>> >
>> > Can you please give me a pointer which shm array change you are
>> referring to?
>>
>> r1824504
>
> Thanks.
>
> What particular breakage are you referring to in the above revision?
> As far as I can tell, the bug does not get fixed for modules that do not use the new field, but it doesn't break them.
Again (I think I commented and this was pointed out on list) that
mod_cluster is broken. However, mod_cluster is broken in the sense
that it rearranges members of the array, and without knowledge of
the future size of the struct, it ultimately fails to succeed.
Modules like mod_ftp and mod_cluster and mod_wsgi and ... these that
assume the role of core functions and believe they know the size of a
future structure, they can't be kept portable. So if earlier comments
about mod_cluster are correct, I am taking this exception as a
mutual-failure scenario of compatibility.
Therefore the proxy balancer scoreboard, for display purposes, should
not cause faults (although the data won't be correct beyond existing
field sizes.) And, for manipulation, no module that deals in the
affected fields will still function. If you believe that
mod_proxy_balancer exists solely for httpd-project implemented
providers, this is not an issue; I continue to vote no position on the
untenable abuse of the 2.4 ABI.
Cheers,
Bill
AW: Binary Breakage (was: svn commit: r1824592 -
/httpd/httpd/branches/2.4.x/STATUS)
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: William A Rowe Jr [mailto:wrowe@rowe-clan.net]
> Gesendet: Dienstag, 20. Februar 2018 22:29
> An: httpd <de...@httpd.apache.org>
> Betreff: Re: Binary Breakage (was: svn commit: r1824592 -
> /httpd/httpd/branches/2.4.x/STATUS)
>
> On Tue, Feb 20, 2018 at 2:57 PM, Ruediger Pluem <rp...@apache.org>
> wrote:
> >
> > On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
> >
> >> Moving a member in a well-defined structure doesn't fall into this
> >> generally accepted change (expanding the length of a struct.)
> >> Consider the shm array change doesn't fall into the "it is just
> >> a longer struct" example.
> >
> > Can you please give me a pointer which shm array change you are
> referring to?
>
> r1824504
Thanks.
What particular breakage are you referring to in the above revision?
As far as I can tell, the bug does not get fixed for modules that do not use the new field, but it doesn't break them.
Regards
Rüdiger
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Tue, Feb 20, 2018 at 2:57 PM, Ruediger Pluem <rp...@apache.org> wrote:
>
> On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
>
>> Moving a member in a well-defined structure doesn't fall into this
>> generally accepted change (expanding the length of a struct.)
>> Consider the shm array change doesn't fall into the "it is just
>> a longer struct" example.
>
> Can you please give me a pointer which shm array change you are referring to?
r1824504
Re: Binary Breakage (was: svn commit: r1824592 -
/httpd/httpd/branches/2.4.x/STATUS)
Posted by Ruediger Pluem <rp...@apache.org>.
On 02/20/2018 09:39 PM, William A Rowe Jr wrote:
> Moving a member in a well-defined structure doesn't fall into this
> generally accepted change (expanding the length of a struct.)
> Consider the shm array change doesn't fall into the "it is just
> a longer struct" example.
>
Can you please give me a pointer which shm array change you are referring to?
Regards
Rüdiger
Re: Binary Breakage (was: svn commit: r1824592 - /httpd/httpd/branches/2.4.x/STATUS)
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Tue, Feb 20, 2018 at 2:24 PM, Ruediger Pluem <rp...@apache.org> wrote:
>
> On 02/20/2018 08:20 PM, William A Rowe Jr wrote:
>
>> In other words, modules from one STABLE release to another ARE binary
>> compatible and do NOT need to be recompiled.
>>
>>
>> This is clearly not true of several recent changes, even though they
>> impact relatively few third party modules.
>>
>> I suggest that since the majority here accepted binary breakage as OK,
>
> I do not think that this is the case. I do not accept this and I don't think the majority here does.
> I am around here for quite a while (effectively shortly before 2.2.x lifted off)
> and it was always allowed to extend structures at the end. This was always seen
> as a minor bump and within the set of allowed changes to a minor release.
> I don't know when these rules were decided, but I grew up with that tribal knowledge
> and we have a long number of precedent cases for these changes on stable branches.
> This is not something that was introduced recently.
> Having said that I admit that it does not match the statement that
> modules do not need to be recompiled once they have been compiled against a major version
> and that they should run without the need to recompile against higher minor versions
> (not necessarily lower ones as they might use added API's).
> Given that the question to me is how we move forward:
>
> 1. We continue as we do now and allow extending structures at the end. We should probably document more prominently that
> copying / allocating / creating public structures is not allowed and if done can require minor version specific recompiling.
Some structures may be extended and always introduce breakage
when a module chooses to copy the structure themself. mod_ftp
and mod_cluster are two examples of this.
In the case of mod_cluster, it depends on the location of members
of the structure, one of which just moved. We perturbed that module
with our member shift. BUT because it assumes it knows the len
of the structure, it made a bad assumption about doing this on the
behalf of a new core module. So it might be a reciprocal breakage.
Similarly, bad on mod_wsgi and mod_ftp and similar for creating
new structs who cannot predict the size; recompile them.
Moving a member in a well-defined structure doesn't fall into this
generally accepted change (expanding the length of a struct.)
Consider the shm array change doesn't fall into the "it is just
a longer struct" example.
The proposed mod_dav case is worse yet. If we accept that the
API demands the submodule pass the struct to mod_dav, there
is no way to distinguish what is going on, and whether this module
was written for 2.4.27 or 2.4.33. Worse 2x yet, we guaranteed that
no source code revision would be required during 2.4.x.
> 2. We stop allowing extending structures at the end on stable branches.
I don't think that is realistic. Adding a new member with an acceptable
NULL/unknown purpose, and adding 3) as noted below, should
generally solve for the case of "module which must be recompiled
on each release".
> 3. Like 1., but we add _sizeof / _copy functions for each public struct as discussed in the other thread, make the
> existence of the functions mandatory for every new public struct and advice module authors that they need to leverage
> them if the want to copy / allocate / create these structures in their modules.
>
> We can do 1., 2. or 3. now or set this as the rule for the next major release.
The advantage to _sizeof/_copy and enforcing the "new member
default must be ok-to-be-NULL" rule means that mod_cluster,
mod_ftp, mod_wsgi and many many other edge-case modules
don't need to be recompiled. This situation can be dramatically
improved in 2.next/3.0. Ensuring _create() accessors are always
provided and demanded to initialize everything that shouldn't
default to NULL is the 100% solve.
The MMN minor does not help. Nothing suggests whether minor
.43 changes the size of any specific member *they* were depending
upon; modules may decide to throw up on load if they were compiled
against .42 and now are loaded into .43. But that doesn't answer the
question of whether something of importance to *that* module has
changed.
Re: Binary Breakage (was: svn commit: r1824592 -
/httpd/httpd/branches/2.4.x/STATUS)
Posted by Ruediger Pluem <rp...@apache.org>.
On 02/20/2018 08:20 PM, William A Rowe Jr wrote:
>
>
> In other words, modules from one STABLE release to another ARE binary
> compatible and do NOT need to be recompiled.
>
>
> This is clearly not true of several recent changes, even though they
> impact relatively few third party modules.
>
> I suggest that since the majority here accepted binary breakage as OK,
I do not think that this is the case. I do not accept this and I don't think the majority here does.
I am around here for quite a while (effectively shortly before 2.2.x lifted off)
and it was always allowed to extend structures at the end. This was always seen
as a minor bump and within the set of allowed changes to a minor release.
I don't know when these rules were decided, but I grew up with that tribal knowledge
and we have a long number of precedent cases for these changes on stable branches.
This is not something that was introduced recently.
Having said that I admit that it does not match the statement that
modules do not need to be recompiled once they have been compiled against a major version
and that they should run without the need to recompile against higher minor versions
(not necessarily lower ones as they might use added API's).
Given that the question to me is how we move forward:
1. We continue as we do now and allow extending structures at the end. We should probably document more prominently that
copying / allocating / creating public structures is not allowed and if done can require minor version specific recompiling.
2. We stop allowing extending structures at the end on stable branches.
3. Like 1., but we add _sizeof / _copy functions for each public struct as discussed in the other thread, make the
existence of the functions mandatory for every new public struct and advice module authors that they need to leverage
them if the want to copy / allocate / create these structures in their modules.
We can do 1., 2. or 3. now or set this as the rule for the next major release.
Regards
Rüdiger