You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Jacob Barrett <ja...@vmware.com> on 2021/01/29 19:38:05 UTC
[Proposal] Geode Native Library Versioning
I would appreciate some feedback on this proposal by the end of day Friday, February 5th, 2021.
TLDR; The proposal is to formalize what has been effectively the status quo for Geode Native release in the current source form and any future binary form.
The Problem
Geode Native, specifically the C++ client, has a versioning problem. This is not a problem with compatibility or versioning between the Geode Native libraries and the Geode Server but rather an issue of compatibility between consumer applications and the Geode Native Libraries. Geode subscribes to the semantic version doctrine [1]. Semver describes API compatibility requirements but is largely silent on ABI, though it can be extended to the ABI. API and ABI compatibility are not the same thing. You can have an API compatible change that breaks the ABI and an ABI compatible change that breaks API. The intersection of the two creates even more complex rules around maintaining compatibility between release. All this in combination makes for really restrictive rules when trying to release even the smallest updates to a library.
For source releases this is really easy, you build what you need with your application and all should be happy. For binary releases this becomes a little harder. Since the consumer isn’t in control of what is built or how it is built, binary incompatibilities between releases can creep in. The difference is in the API vs ABI compatibility and while more pronounced in some languages it can exist in all. For example, one could create an ABI incompatibility in Java by compiling class files with a newer version of the runtime than the previous release. Those new classes, while API compatible, would not be loadable by the older runtime and classes thus making them ABI incompatible. In C++ not only does the runtime version play a role but the physical layout in memory, name mangle strategies, optimizations and other things can cause ABI incompatibility [2].
Geode Native has not maintained ABI compatibility between minor releases. Minor releases have included changes to exported types and methods changing. In the upcoming release of 1.14 there are a few incompatible changes around a new virtual method added to a base type. While the C++ language does not define how compilers and linkers should treat such changes they almost universally result in layout changes of the vtable and thus ABI incompatibilities.
The results of these incompatibilities at the ABI level are undefined and hard to diagnose. In a pinch if a consumer placed a patched library into their application that had an ABI compatibility issue it may appear to work but over time could exhibit odd behavior. Best case scenario it causes the application to exit but worst case it silently executes the wrong virtual methods resulting in the loss of data.
ABI Versioning Options
We must communicate ABI versioning to consumers of the library. Before we can do that we must decide on ABI versioning rules.
Semantic Versioning for ABI
This is probably the most restrictive of our options. If the ABI is included in semantic versioning then no changes can be made to types with virtual methods. For example, the recent change to add expected domain names to PDX types would not have been possible until a major release. This change resulted in the change to a base type virtual method, thus a vtable change and an ABI incompatible change. The benefit to consumers is that they never need to recompile their applications for minor or patch updates unless taking advantage of non-breaking API compatible changes.
No ABI Versioning
If ABI is not considered as part of the semantic versioning scheme then every release expects a complete recompile from all applications. This is an easy rule for consumers to follow but isn’t ideal for minor patches where it is likely a drop in binary could be possible and certainly less disruptive than a full recompile.
Hybrid Semantic Versioning
We could communicate some custom rule where only patch level semantic versioned releases are ABI compatible and any major or minor release requires a recompile. I believe we are effectively operating in this space but without explicit intent, though no care is being taken to verify that even patch releases have been ABI compatible.
Proposal
I propose that we adopt a hybrid semantic versioning convention that establishes intent to maintain ABI compatibility only between patch releases.
Major releases will contain both API and ABI incompatible changes. The library may not be replaced without code changes and recompilation. Changes to API and ABI will be documented in a change log for easy reference.
Minor releases may contain API compatible changes and should be assumed to have ABI incompatible changes. The library may not be replaced without recompilation but code changes are not necessary. Changes to API and ABI will be documented in a change log for easy reference.
Patch releases may contain API and ABI compatible changes. The library may be replaced without recompilation.
Enforcement
Enforcement of ABI compatibility rules is complex. Each compiler and platform have their own binary rules so simply taking one platform’s or compiler’s rules isn’t going to catch all cases but it is a starting point.
Strict Review of Public Headers
Putting more care into review of changes to public headers is a starting point. Almost any change to a public header is by definition an API change. Some of those changes are going to have an effect on the ABI as well. Analysing those changes for their ABI impact manually prior to acceptance should minimize the chances of an incompatible ABI breaking changes entering a release at the wrong time.
ABI Comparison Tools
Tools exist to compare ABIs between two versions of a library. We can include these tools in our CI pipelines to identify any unintended ABI changes that may be merged in.
Library Version Check
Consider implementation of some form of library version checking that would fail early at runtime if the potential exists for ABI incompatibility with linked applications. Solutions such as versioned library naming, versioned namespacing, or version constant checking should be investigated.
Future
Future work should move towards a more stable ABI where the ABI could follow semantic versioning.
Stable C-style ABI
Move towards a more stable C-style ABI for the layer between the language, C++, .NET, etc., to avoid the instability of a C++ mangled ABI. This also moves us closer to supporting more languages that support C library bindings by treating even C++ as a language bound to the library via a C-style ABI.
Decreased Reliance on Virtual Methods
The most dangerous and restrictive ABI changes are to vtables for virtual methods. If we can remove virtual methods from the public API entirely this would drastically improve our ability to maintain ABI compatibility even with API changes. The RegionService class and its derivatives should be replaced with something more ABI compatible.
Increase Header Only Methods
Increasing our reliance on header only implementation, similar to Boost, STL, and other popular libraries, would help significantly. If most of the implementation was compiled into the application leaving only a small layer of ABI to a library reduces the surface area to be concerned about ABI changes. Between releases the thin library could be more ABI compatible without recompilation, especially if that layer was agnostic to nearly all the header only defined types. Such an ABI would likely be more compatible as a binding layer for other languages as well.
[1] https://semver.org<https://semver.org/>
[2] https://www.youtube.com/watch?v=k9PLRAnnEmE
Thanks,
Jake
Re: [Proposal] Geode Native Library Versioning
Posted by Jacob Barrett <ja...@vmware.com>.
Thanks for the feedback. The corresponding README and CONTRIBUTING docs have been updated to reflect this proposal.
> On Feb 2, 2021, at 4:41 AM, Mario Salazar de Torres <ma...@est.tech> wrote:
>
> Thanks for pointing out this issue. A few pointers on your proposal:
>
> * On my opinion the hybrid approach is the most practical one. For our case, we compile from the source each release, but it might seem reasonable that patch releases maintain both ABI and API compatibility.
> * To make sure API and ABI compatibility it would be good to write down the compatibility rules in the documentation (i.e: CONTRIBUTING.md)
> * Regarding future changes, I agree that there are lots of virtual methods that might not need to be virtual and might be good to re-think the public API for the next major release to improve it.
> * Also, a note on the fact that we are exporting more symbols than needed. For most cases that's done this way to be able to test some features.
> So, I'd say both things are closely related and to reduce the exported symbols, as well as the number of virtual methods we might need to work on the testing approach.
>
> BR/
> Mario.
> ________________________________
> From: Jacob Barrett <ja...@vmware.com>
> Sent: Monday, February 1, 2021 4:34 PM
> To: dev@geode.apache.org <de...@geode.apache.org>
> Subject: Re: [Proposal] Geode Native Library Versioning
>
>
> On Jan 29, 2021, at 3:47 PM, Dan Smith <da...@vmware.com>> wrote:
> I do think at least implementing some automated checking for whatever compatibility we intend to provide is a good idea.
>
> I have a branch with a test using Abigail [1]. This branch depends on merging of a CI branch. It was actually through this branch that it was clear we have not been good about preserving any ABI compatibility between releases. I also found that we are exporting more symbols than we should in our library. All these need to be addressed over time. Baby steps I supposed…
>
> [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceware.org%2Flibabigail%2F&data=04%7C01%7Cjabarrett%40vmware.com%7Cdef3066d7ed24b9b36c708d8c777cc99%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637478664662167370%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ccHSB7DnIqLP02Jv7h73SvYQ9uEcIWozFEQViA48egU%3D&reserved=0
>
> -Jake
>
Re: [Proposal] Geode Native Library Versioning
Posted by Mario Salazar de Torres <ma...@est.tech>.
Thanks for pointing out this issue. A few pointers on your proposal:
* On my opinion the hybrid approach is the most practical one. For our case, we compile from the source each release, but it might seem reasonable that patch releases maintain both ABI and API compatibility.
* To make sure API and ABI compatibility it would be good to write down the compatibility rules in the documentation (i.e: CONTRIBUTING.md)
* Regarding future changes, I agree that there are lots of virtual methods that might not need to be virtual and might be good to re-think the public API for the next major release to improve it.
* Also, a note on the fact that we are exporting more symbols than needed. For most cases that's done this way to be able to test some features.
So, I'd say both things are closely related and to reduce the exported symbols, as well as the number of virtual methods we might need to work on the testing approach.
BR/
Mario.
________________________________
From: Jacob Barrett <ja...@vmware.com>
Sent: Monday, February 1, 2021 4:34 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [Proposal] Geode Native Library Versioning
On Jan 29, 2021, at 3:47 PM, Dan Smith <da...@vmware.com>> wrote:
I do think at least implementing some automated checking for whatever compatibility we intend to provide is a good idea.
I have a branch with a test using Abigail [1]. This branch depends on merging of a CI branch. It was actually through this branch that it was clear we have not been good about preserving any ABI compatibility between releases. I also found that we are exporting more symbols than we should in our library. All these need to be addressed over time. Baby steps I supposed…
[1] https://sourceware.org/libabigail/
-Jake
Re: [Proposal] Geode Native Library Versioning
Posted by Jacob Barrett <ja...@vmware.com>.
On Jan 29, 2021, at 3:47 PM, Dan Smith <da...@vmware.com>> wrote:
I do think at least implementing some automated checking for whatever compatibility we intend to provide is a good idea.
I have a branch with a test using Abigail [1]. This branch depends on merging of a CI branch. It was actually through this branch that it was clear we have not been good about preserving any ABI compatibility between releases. I also found that we are exporting more symbols than we should in our library. All these need to be addressed over time. Baby steps I supposed…
[1] https://sourceware.org/libabigail/
-Jake
Re: [Proposal] Geode Native Library Versioning
Posted by Dan Smith <da...@vmware.com>.
ABI compatibility with the java layer has come up before [1]. At the time we decided to keep binary compatibility for the Java code.
That said, it's not common to break binary without breaking source compatibility in Java. C++ is a different animal, so I can see wanting to do something different there. I do think at least implementing some automated checking for whatever compatibility we intend to provide is a good idea.
[1] https://lists.apache.org/x/thread.html/ebdeabc24ce0ff418427021dfbbd45aceb51939f25de4aba7bb5c97b@%3Cdev.geode.apache.org%3E
________________________________
From: Jacob Barrett <ja...@vmware.com>
Sent: Friday, January 29, 2021 12:00 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [Proposal] Geode Native Library Versioning
** sorry for the trashed formatting originally **
I would appreciate some feedback on this proposal by the end of day Friday, February 5th, 2021.
TLDR; The proposal is to formalize what has been effectively the status quo for Geode Native release in the current source form and any future binary form.
= The Problem =
Geode Native, specifically the C++ client, has a versioning problem. This is not a problem with compatibility or versioning between the Geode Native libraries and the Geode Server but rather an issue of compatibility between consumer applications and the Geode Native Libraries. Geode subscribes to the semantic version doctrine [1]. Semver describes API compatibility requirements but is largely silent on ABI, though it can be extended to the ABI. API and ABI compatibility are not the same thing. You can have an API compatible change that breaks the ABI and an ABI compatible change that breaks API. The intersection of the two creates even more complex rules around maintaining compatibility between release. All this in combination makes for really restrictive rules when trying to release even the smallest updates to a library.
For source releases this is really easy, you build what you need with your application and all should be happy. For binary releases this becomes a little harder. Since the consumer isn’t in control of what is built or how it is built, binary incompatibilities between releases can creep in. The difference is in the API vs ABI compatibility and while more pronounced in some languages it can exist in all. For example, one could create an ABI incompatibility in Java by compiling class files with a newer version of the runtime than the previous release. Those new classes, while API compatible, would not be loadable by the older runtime and classes thus making them ABI incompatible. In C++ not only does the runtime version play a role but the physical layout in memory, name mangle strategies, optimizations and other things can cause ABI incompatibility [2].
Geode Native has not maintained ABI compatibility between minor releases. Minor releases have included changes to exported types and methods changing. In the upcoming release of 1.14 there are a few incompatible changes around a new virtual method added to a base type. While the C++ language does not define how compilers and linkers should treat such changes they almost universally result in layout changes of the vtable and thus ABI incompatibilities.
The results of these incompatibilities at the ABI level are undefined and hard to diagnose. In a pinch if a consumer placed a patched library into their application that had an ABI compatibility issue it may appear to work but over time could exhibit odd behavior. Best case scenario it causes the application to exit but worst case it silently executes the wrong virtual methods resulting in the loss of data.
= ABI Versioning Options =
We must communicate ABI versioning to consumers of the library. Before we can do that we must decide on ABI versioning rules.
== Semantic Versioning for ABI ==
This is probably the most restrictive of our options. If the ABI is included in semantic versioning then no changes can be made to types with virtual methods. For example, the recent change to add expected domain names to PDX types would not have been possible until a major release. This change resulted in the change to a base type virtual method, thus a vtable change and an ABI incompatible change. The benefit to consumers is that they never need to recompile their applications for minor or patch updates unless taking advantage of non-breaking API compatible changes.
== No ABI Versioning ==
If ABI is not considered as part of the semantic versioning scheme then every release expects a complete recompile from all applications. This is an easy rule for consumers to follow but isn’t ideal for minor patches where it is likely a drop in binary could be possible and certainly less disruptive than a full recompile.
== Hybrid Semantic Versioning ==
We could communicate some custom rule where only patch level semantic versioned releases are ABI compatible and any major or minor release requires a recompile. I believe we are effectively operating in this space but without explicit intent, though no care is being taken to verify that even patch releases have been ABI compatible.
= Proposal =
I propose that we adopt a hybrid semantic versioning convention that establishes intent to maintain ABI compatibility only between patch releases.
Major releases will contain both API and ABI incompatible changes. The library may not be replaced without code changes and recompilation. Changes to API and ABI will be documented in a change log for easy reference.
Minor releases may contain API compatible changes and should be assumed to have ABI incompatible changes. The library may not be replaced without recompilation but code changes are not necessary. Changes to API and ABI will be documented in a change log for easy reference.
Patch releases may contain API and ABI compatible changes. The library may be replaced without recompilation.
== Enforcement ==
Enforcement of ABI compatibility rules is complex. Each compiler and platform have their own binary rules so simply taking one platform’s or compiler’s rules isn’t going to catch all cases but it is a starting point.
=== Strict Review of Public Headers ===
Putting more care into review of changes to public headers is a starting point. Almost any change to a public header is by definition an API change. Some of those changes are going to have an effect on the ABI as well. Analysing those changes for their ABI impact manually prior to acceptance should minimize the chances of an incompatible ABI breaking changes entering a release at the wrong time.
=== ABI Comparison Tools ===
Tools exist to compare ABIs between two versions of a library. We can include these tools in our CI pipelines to identify any unintended ABI changes that may be merged in.
=== Library Version Check ===
Consider implementation of some form of library version checking that would fail early at runtime if the potential exists for ABI incompatibility with linked applications. Solutions such as versioned library naming, versioned namespacing, or version constant checking should be investigated.
== Future ==
Future work should move towards a more stable ABI where the ABI could follow semantic versioning.
=== Stable C-style ABI ===
Move towards a more stable C-style ABI for the layer between the language, C++, .NET, etc., to avoid the instability of a C++ mangled ABI. This also moves us closer to supporting more languages that support C library bindings by treating even C++ as a language bound to the library via a C-style ABI.
=== Decreased Reliance on Virtual Methods ===
The most dangerous and restrictive ABI changes are to vtables for virtual methods. If we can remove virtual methods from the public API entirely this would drastically improve our ability to maintain ABI compatibility even with API changes. The RegionService class and its derivatives should be replaced with something more ABI compatible.
=== Increase Header Only Methods ===
Increasing our reliance on header only implementation, similar to Boost, STL, and other popular libraries, would help significantly. If most of the implementation was compiled into the application leaving only a small layer of ABI to a library reduces the surface area to be concerned about ABI changes. Between releases the thin library could be more ABI compatible without recompilation, especially if that layer was agnostic to nearly all the header only defined types. Such an ABI would likely be more compatible as a binding layer for other languages as well.
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsemver.org%2F&data=04%7C01%7Cdasmith%40vmware.com%7C3a55ad37e1ac426fec0608d8c4908220%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637475472240110896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1MGSN9RN1khvsZKhorRAtIsJTNI1HXzJNypPzApGR8A%3D&reserved=0
[2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dk9PLRAnnEmE&data=04%7C01%7Cdasmith%40vmware.com%7C3a55ad37e1ac426fec0608d8c4908220%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637475472240110896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4fF79k26bySJ8MbIhQPfL%2BWd1egajT1dt0O6hlrI5n0%3D&reserved=0
Thanks,
Jake
Re: [Proposal] Geode Native Library Versioning
Posted by Jacob Barrett <ja...@vmware.com>.
** sorry for the trashed formatting originally **
I would appreciate some feedback on this proposal by the end of day Friday, February 5th, 2021.
TLDR; The proposal is to formalize what has been effectively the status quo for Geode Native release in the current source form and any future binary form.
= The Problem =
Geode Native, specifically the C++ client, has a versioning problem. This is not a problem with compatibility or versioning between the Geode Native libraries and the Geode Server but rather an issue of compatibility between consumer applications and the Geode Native Libraries. Geode subscribes to the semantic version doctrine [1]. Semver describes API compatibility requirements but is largely silent on ABI, though it can be extended to the ABI. API and ABI compatibility are not the same thing. You can have an API compatible change that breaks the ABI and an ABI compatible change that breaks API. The intersection of the two creates even more complex rules around maintaining compatibility between release. All this in combination makes for really restrictive rules when trying to release even the smallest updates to a library.
For source releases this is really easy, you build what you need with your application and all should be happy. For binary releases this becomes a little harder. Since the consumer isn’t in control of what is built or how it is built, binary incompatibilities between releases can creep in. The difference is in the API vs ABI compatibility and while more pronounced in some languages it can exist in all. For example, one could create an ABI incompatibility in Java by compiling class files with a newer version of the runtime than the previous release. Those new classes, while API compatible, would not be loadable by the older runtime and classes thus making them ABI incompatible. In C++ not only does the runtime version play a role but the physical layout in memory, name mangle strategies, optimizations and other things can cause ABI incompatibility [2].
Geode Native has not maintained ABI compatibility between minor releases. Minor releases have included changes to exported types and methods changing. In the upcoming release of 1.14 there are a few incompatible changes around a new virtual method added to a base type. While the C++ language does not define how compilers and linkers should treat such changes they almost universally result in layout changes of the vtable and thus ABI incompatibilities.
The results of these incompatibilities at the ABI level are undefined and hard to diagnose. In a pinch if a consumer placed a patched library into their application that had an ABI compatibility issue it may appear to work but over time could exhibit odd behavior. Best case scenario it causes the application to exit but worst case it silently executes the wrong virtual methods resulting in the loss of data.
= ABI Versioning Options =
We must communicate ABI versioning to consumers of the library. Before we can do that we must decide on ABI versioning rules.
== Semantic Versioning for ABI ==
This is probably the most restrictive of our options. If the ABI is included in semantic versioning then no changes can be made to types with virtual methods. For example, the recent change to add expected domain names to PDX types would not have been possible until a major release. This change resulted in the change to a base type virtual method, thus a vtable change and an ABI incompatible change. The benefit to consumers is that they never need to recompile their applications for minor or patch updates unless taking advantage of non-breaking API compatible changes.
== No ABI Versioning ==
If ABI is not considered as part of the semantic versioning scheme then every release expects a complete recompile from all applications. This is an easy rule for consumers to follow but isn’t ideal for minor patches where it is likely a drop in binary could be possible and certainly less disruptive than a full recompile.
== Hybrid Semantic Versioning ==
We could communicate some custom rule where only patch level semantic versioned releases are ABI compatible and any major or minor release requires a recompile. I believe we are effectively operating in this space but without explicit intent, though no care is being taken to verify that even patch releases have been ABI compatible.
= Proposal =
I propose that we adopt a hybrid semantic versioning convention that establishes intent to maintain ABI compatibility only between patch releases.
Major releases will contain both API and ABI incompatible changes. The library may not be replaced without code changes and recompilation. Changes to API and ABI will be documented in a change log for easy reference.
Minor releases may contain API compatible changes and should be assumed to have ABI incompatible changes. The library may not be replaced without recompilation but code changes are not necessary. Changes to API and ABI will be documented in a change log for easy reference.
Patch releases may contain API and ABI compatible changes. The library may be replaced without recompilation.
== Enforcement ==
Enforcement of ABI compatibility rules is complex. Each compiler and platform have their own binary rules so simply taking one platform’s or compiler’s rules isn’t going to catch all cases but it is a starting point.
=== Strict Review of Public Headers ===
Putting more care into review of changes to public headers is a starting point. Almost any change to a public header is by definition an API change. Some of those changes are going to have an effect on the ABI as well. Analysing those changes for their ABI impact manually prior to acceptance should minimize the chances of an incompatible ABI breaking changes entering a release at the wrong time.
=== ABI Comparison Tools ===
Tools exist to compare ABIs between two versions of a library. We can include these tools in our CI pipelines to identify any unintended ABI changes that may be merged in.
=== Library Version Check ===
Consider implementation of some form of library version checking that would fail early at runtime if the potential exists for ABI incompatibility with linked applications. Solutions such as versioned library naming, versioned namespacing, or version constant checking should be investigated.
== Future ==
Future work should move towards a more stable ABI where the ABI could follow semantic versioning.
=== Stable C-style ABI ===
Move towards a more stable C-style ABI for the layer between the language, C++, .NET, etc., to avoid the instability of a C++ mangled ABI. This also moves us closer to supporting more languages that support C library bindings by treating even C++ as a language bound to the library via a C-style ABI.
=== Decreased Reliance on Virtual Methods ===
The most dangerous and restrictive ABI changes are to vtables for virtual methods. If we can remove virtual methods from the public API entirely this would drastically improve our ability to maintain ABI compatibility even with API changes. The RegionService class and its derivatives should be replaced with something more ABI compatible.
=== Increase Header Only Methods ===
Increasing our reliance on header only implementation, similar to Boost, STL, and other popular libraries, would help significantly. If most of the implementation was compiled into the application leaving only a small layer of ABI to a library reduces the surface area to be concerned about ABI changes. Between releases the thin library could be more ABI compatible without recompilation, especially if that layer was agnostic to nearly all the header only defined types. Such an ABI would likely be more compatible as a binding layer for other languages as well.
[1] https://semver.org
[2] https://www.youtube.com/watch?v=k9PLRAnnEmE
Thanks,
Jake