Positioning Celix against CppMicroservices, a C++ OSGi implementation

[Michael de Lang <oi...@apache.org>]

It seems we've (or at least I have) missed the already existing CppMicroservices (https://github.com/CppMicroServices/CppMicroServices) framework, which aims to implement OSGi for C++14. Given our current discussions/efforts into providing C++ API for Celix, we have a couple of critical questions we need to ask ourselves.

I've tried to assess the current situation, feel free to let me know if I made omissions:
* Celix currently provides a partial implementation of OSGi, of which some parts are not (yet?) implemented in CppMicroservices (pubsub, RSA) and some parts are not in OSGi at all (etcdlib, iovec implementation). 
* Both Celix and CppMicroservices both have one big commercial user (AFAIK): Celix has Thales and CppMicroservices has German Cancer Research Center (DKFZ)
* Both Celix and CppMicroservices have similar order of magnitude of activities outside of the main contributors (watches/stars/forks)
* There are currently two approaches at providing C++-esque support for what Celix stands for. The experimental C++ API for Celix as can be seen in Pepijn's PR (https://github.com/apache/celix/pull/259) and a more revolutionary approach where OSGi is only used as an inspiration and adds things like a clear threading model; Cppelix (https://github.com/volt-software/cppelix).

The questions we must then ask ourselves are as follows:

* Do we believe a C implementation of the OSGi specification is a worthwhile addition alongside other implementations such as CppMicroservices?
* Do we think it is worthwhile to offer a C++ API around Celix if other implementations offer a native C++ API?

If we honestly believe that providing a C++ API is the way to go, perhaps we should consider the following two options:
* Joining forces/forking and implementing Celix functionality like the stateless pubsub/RSA components in CppMicroservices
* Creating an evolution of the OSGi spec, such as Cppelix is an attempt at, and focusing on that.
* Perhaps a third option that I haven't thought of yet

Don't get me wrong, Celix is a product with lots of functionality in it that isn't necessarily in the OSGi and is useful. However, I truly think that the reality is that there are serious attempts in the world around us in making similar products. We would do well to formulate a strategy with which we can respond to these circumstances.

Re: Positioning Celix against CppMicroservices, a C++ OSGi implementation

[Michael de Lang <oi...@apache.org>]

Hey Pepijn,

Thanks for taking the time to thoroughly answer.

On 2020/08/16 20:21:47, Pepijn Noltes <pe...@gmail.com> wrote: 
> Hi Michael,
> 
> 
> On Sat, Aug 15, 2020 at 10:41 PM Michael de Lang <oi...@apache.org> wrote:
> >
> > It seems we've (or at least I have) missed the already existing CppMicroservices (https://github.com/CppMicroServices/CppMicroServices) framework, which aims to implement OSGi for C++14. Given our current discussions/efforts into providing C++ API for Celix, we have a couple of critical questions we need to ask ourselves.
> 
> I know CppMicroservices. Back in - I think - 2012 we talked with the
> lead developer of CppMircoservices about a possible Native OSGI spec
> and a shared API.
> And although there was some effort to get this up and running this
> eventually fizzled out. IMO mainly because it is difficult to define a
> C and C++ API only library.
> 
> The main differences between CppMicroservices (then) was that Celix
> uses C and CppMicroservices C++11 (which was fairly new then) and
> Celix more directly followed the OSGi specification.
> 

Ah, yeah, it's quite a big difference and it would require both sides to want to implement more or less the same thing in both languages. Would you say that the current status of CppMicroservices still seems like it adheres less to the specification? The bundles listen on their website (http://cppmicroservices.org/) as well as the ability to make zips and embed resources (http://docs.cppmicroservices.org/en/stable/framework/doc/resources.html) make it look like it follows it relatively well. Or was the difference never that pronounced?

> >
> > I've tried to assess the current situation, feel free to let me know if I made omissions:
> > * Celix currently provides a partial implementation of OSGi, of which some parts are not (yet?) implemented in CppMicroservices (pubsub, RSA) and some parts are not in OSGi at all (etcdlib, iovec implementation).
> > * Both Celix and CppMicroservices both have one big commercial user (AFAIK): Celix has Thales and CppMicroservices has German Cancer Research Center (DKFZ)
> > * Both Celix and CppMicroservices have similar order of magnitude of activities outside of the main contributors (watches/stars/forks)
> > * There are currently two approaches at providing C++-esque support for what Celix stands for. The experimental C++ API for Celix as can be seen in Pepijn's PR (https://github.com/apache/celix/pull/259) and a more revolutionary approach where OSGi is only used as an inspiration and adds things like a clear threading model; Cppelix (https://github.com/volt-software/cppelix).
> >
> > The questions we must then ask ourselves are as follows:
> >
> > * Do we believe a C implementation of the OSGi specification is a worthwhile addition alongside other implementations such as CppMicroservices?
> > * Do we think it is worthwhile to offer a C++ API around Celix if other implementations offer a native C++ API?
> 
> Good point. My idea was always to create a Celix C++ library which
> also offers a C API. This library can co-exist - at least for a while
> - with the C library and create an option to seamlessly use Celix C
> bundles in a possible future Celix C++ framework. This is also a big
> reason why I think a C++ Celix library next to CppMicroservices has
> value.
> 
> I think it is important to support a C api for two reasons:
> 1) (very simple) We already build a lot of Celix C Bundles (both on
> Apache Celix and commercial). This then creates an option to gradually
> (bundle per bundle) move from C to C++ (if this is desirable).
> 2) To support modularization of legacy C systems with a low entry
> point to start. This is maybe a stress, but I still believe Apache
> Celix is a good solution to modularize complex legacy system build
> with C. C is still heavily used and has two very big benefits over C++
> ... it is a very stable and small language.

Hmm, while I do think that celix offers a lot for the C language, I'm still on the fence when it comes to providing both a C and a C++ API from the same library. Rather, I have the suspicion that making them compatible on the network layer (so seamless use of RSA and  pubsub) is all that is needed. This would mean two separate libraries, but both catered well to the language they aim for.

Also, while C is stable and 'small', mastering the language takes a long time: the language doesn't prevent you at all from shooting yourself in the foot ;)

> 
> >
> > If we honestly believe that providing a C++ API is the way to go, perhaps we should consider the following two options:
> > * Joining forces/forking and implementing Celix functionality like the stateless pubsub/RSA components in CppMicroservices
> > * Creating an evolution of the OSGi spec, such as Cppelix is an attempt at, and focusing on that.
> > * Perhaps a third option that I haven't thought of yet
> >
> > Don't get me wrong, Celix is a product with lots of functionality in it that isn't necessarily in the OSGi and is useful. However, I truly think that the reality is that there are serious attempts in the world around us in making similar products. We would do well to formulate a strategy with which we can respond to these circumstances.
> 
> A good point. I think you are correct that if we just implement a
> C++11/14/17 API this has no real added value compared to
> CppMicroservices.
> 
> But I think cppelix is an interesting starting point for a possible
> future of Apache Celix. For a few different reasons
> 1) The OSGi framework spec does - almost - nothing to make multi
> threading programming more manageable, IMO a future Celix should
> address this. We are struggling with MT in the current Celix
> libraries/bundles and this adds unnecessary unstability and
> complexity. The cppelix framework tries to address this.
> 2) C++20. I consider the "pre C++11", C++11/14/17 and C++20  different
> languages, they change so much of the used programming paradigms that
> read as different languages. So a future Celix could be a C++20
> framework with a C++20 API and a C API. The C++20 API can then use
> concepts, modules, "constexpr everything", coroutines, etc. This IMO
> would really create an added value for Celix, specifically if you can
> "just" start with C for the next few years and step over to C++20 when
> this is broader supported and more well known.
> 
> One note though is that I would prefer that Celix tries to follow the
> concepts and especially the nomenclature of OSGi when possible. This
> should keep Celix being familiar for people with an OSGi background
> and should make it possible to very directly implement  the OSGi
> compendium/enterprise spec parts.
> How I currently think about this, is maybe start with the OSGi Promise
> (we already started in this) & Pushstream spec and integrate this in
> the C++ framework library.
> So for example a service registration is returned as a promise to a
> service registration, chaining this promise can be used to do post
> registration actions on a well defined thread or thread pool.
> PushStream can be an integral part of the Framework and serve as a
> replacement for bundle listener, service tracker, listener hooks and
> even the event admin and a way to handle events in a thread confined
> manner.
> 
> Greetings,
> Pepijn
> 

I generally agree with your points and you raise a good one as well: coming from the OSGi specification and moving to another language, libraries that lower the learning curve would be favourable. However, and this is specifically targeted to the OSGi Promise & Pushstream spec, some parts of the spec are, IMO, very outdated. The trend in modern languages to add for example coroutines (or more commonly known as async/await), is in the general case a superior solution. Although promises are easy to use with coroutines, it should generally be discouraged to use promises without them. I can't really think of any situation where a promise on its own would be preferable. However, this would obviously cause a disconnect between what a user from another OSGi implementation is familiar with. Moreover, it is a deviation from the OSGi spec.

I guess what I'm trying to say is, how much should one really go for familiarity over improvements? I don't have an answer yet, I need some more time to think about it. For a C library I can imagine staying with the OSGi spec, as while coroutines libraries are available for C, I don't know if I would personally use them in C. For a pure C++20 library, I would prefer to encourage users to use coroutines on top of discouraging users to use standalone promises. But then making a C API around that C++20 library would become rather difficult. It is rather difficult to fully understand the consequences of these choices.

Yours sincerely,
Michael

Re: Positioning Celix against CppMicroservices, a C++ OSGi implementation

[Pepijn Noltes <pe...@gmail.com>]

Hi Michael,


On Sat, Aug 15, 2020 at 10:41 PM Michael de Lang <oi...@apache.org> wrote:
>
> It seems we've (or at least I have) missed the already existing CppMicroservices (https://github.com/CppMicroServices/CppMicroServices) framework, which aims to implement OSGi for C++14. Given our current discussions/efforts into providing C++ API for Celix, we have a couple of critical questions we need to ask ourselves.

I know CppMicroservices. Back in - I think - 2012 we talked with the
lead developer of CppMircoservices about a possible Native OSGI spec
and a shared API.
And although there was some effort to get this up and running this
eventually fizzled out. IMO mainly because it is difficult to define a
C and C++ API only library.

The main differences between CppMicroservices (then) was that Celix
uses C and CppMicroservices C++11 (which was fairly new then) and
Celix more directly followed the OSGi specification.

>
> I've tried to assess the current situation, feel free to let me know if I made omissions:
> * Celix currently provides a partial implementation of OSGi, of which some parts are not (yet?) implemented in CppMicroservices (pubsub, RSA) and some parts are not in OSGi at all (etcdlib, iovec implementation).
> * Both Celix and CppMicroservices both have one big commercial user (AFAIK): Celix has Thales and CppMicroservices has German Cancer Research Center (DKFZ)
> * Both Celix and CppMicroservices have similar order of magnitude of activities outside of the main contributors (watches/stars/forks)
> * There are currently two approaches at providing C++-esque support for what Celix stands for. The experimental C++ API for Celix as can be seen in Pepijn's PR (https://github.com/apache/celix/pull/259) and a more revolutionary approach where OSGi is only used as an inspiration and adds things like a clear threading model; Cppelix (https://github.com/volt-software/cppelix).
>
> The questions we must then ask ourselves are as follows:
>
> * Do we believe a C implementation of the OSGi specification is a worthwhile addition alongside other implementations such as CppMicroservices?
> * Do we think it is worthwhile to offer a C++ API around Celix if other implementations offer a native C++ API?

Good point. My idea was always to create a Celix C++ library which
also offers a C API. This library can co-exist - at least for a while
- with the C library and create an option to seamlessly use Celix C
bundles in a possible future Celix C++ framework. This is also a big
reason why I think a C++ Celix library next to CppMicroservices has
value.

I think it is important to support a C api for two reasons:
1) (very simple) We already build a lot of Celix C Bundles (both on
Apache Celix and commercial). This then creates an option to gradually
(bundle per bundle) move from C to C++ (if this is desirable).
2) To support modularization of legacy C systems with a low entry
point to start. This is maybe a stress, but I still believe Apache
Celix is a good solution to modularize complex legacy system build
with C. C is still heavily used and has two very big benefits over C++
... it is a very stable and small language.

>
> If we honestly believe that providing a C++ API is the way to go, perhaps we should consider the following two options:
> * Joining forces/forking and implementing Celix functionality like the stateless pubsub/RSA components in CppMicroservices
> * Creating an evolution of the OSGi spec, such as Cppelix is an attempt at, and focusing on that.
> * Perhaps a third option that I haven't thought of yet
>
> Don't get me wrong, Celix is a product with lots of functionality in it that isn't necessarily in the OSGi and is useful. However, I truly think that the reality is that there are serious attempts in the world around us in making similar products. We would do well to formulate a strategy with which we can respond to these circumstances.

A good point. I think you are correct that if we just implement a
C++11/14/17 API this has no real added value compared to
CppMicroservices.

But I think cppelix is an interesting starting point for a possible
future of Apache Celix. For a few different reasons
1) The OSGi framework spec does - almost - nothing to make multi
threading programming more manageable, IMO a future Celix should
address this. We are struggling with MT in the current Celix
libraries/bundles and this adds unnecessary unstability and
complexity. The cppelix framework tries to address this.
2) C++20. I consider the "pre C++11", C++11/14/17 and C++20  different
languages, they change so much of the used programming paradigms that
read as different languages. So a future Celix could be a C++20
framework with a C++20 API and a C API. The C++20 API can then use
concepts, modules, "constexpr everything", coroutines, etc. This IMO
would really create an added value for Celix, specifically if you can
"just" start with C for the next few years and step over to C++20 when
this is broader supported and more well known.

One note though is that I would prefer that Celix tries to follow the
concepts and especially the nomenclature of OSGi when possible. This
should keep Celix being familiar for people with an OSGi background
and should make it possible to very directly implement  the OSGi
compendium/enterprise spec parts.
How I currently think about this, is maybe start with the OSGi Promise
(we already started in this) & Pushstream spec and integrate this in
the C++ framework library.
So for example a service registration is returned as a promise to a
service registration, chaining this promise can be used to do post
registration actions on a well defined thread or thread pool.
PushStream can be an integral part of the Framework and serve as a
replacement for bundle listener, service tracker, listener hooks and
even the event admin and a way to handle events in a thread confined
manner.

Greetings,
Pepijn