Publisher / subscriber implementation for Celix

[Roy Lenferink <le...@gmail.com>]

Hi everyone,

I'm Roy Lenferink, a software engineering student at Saxion University of
Applied Sciences.
For my 3rd year I'm having an internship at Thales Nederland.

Over the past few months I've been working together with Pepijn Noltes,
Erjan Altena and
Gabriele Ricciardi on a (secure) publisher / subscriber implementation,
which is based on
the concepts of the remote service admin (i.e. rsa / topology / discovery
pattern).

Publishers are senders of data, subscribers can receive data. Publishers
can publish/send
data to certain channels (called 'topics' further on), subscribers can
subscribe to these
topics. For every topic a publisher service is created by the pubsub admin.
This publisher
is announced through etcd. So etcd is used for discovery of the publishers.
Subscribers are
also registered as a service by the pubsub admin and will watch etcd for
changes and when
a new publisher is announced, the subscriber will check if the topic
matches its interests.
If the subscriber is interested in/subscribed to a certain topic, a
connection between
publisher and subscriber will be instantiated by the pubsub admin.

The dfi library is used for message serialization. The publisher /
subscriber implementation
will arrange that every message which will be send gets an unique id.

For communication between publishers and subscribers UDP and ZeroMQ can be
used. When using
ZeroMQ it's also possible to setup a secure connection to encrypt the
traffic being send
between publishers and subscribers. This connection can be secured with
ZeroMQ by using
a curve25519 key pair per topic.

The publisher/subscriber implementation supports sending of a single
message and sending
of multipart messages.

Soon I will create a JIRA issue together with a pull request to get this
part included within Celix.

If you have any further questions, just reply to this mail!

Roy

Re: Publisher / subscriber implementation for Celix

[Pepijn Noltes <pe...@gmail.com>]

Hi,

Welcome to the Celix community Roy :)

On Mon, Dec 19, 2016 at 12:00 PM Gabriele Ricciardi <
lele.ricciardi@gmail.com> wrote:

> Thanks for your contribution, we look forward to integrate your pull
> request! :-)
>
> Gabriele
>
> 2016-12-19 11:58 GMT+01:00 Gabriele Ricciardi <le...@gmail.com>:
>
> > Hi Roy,
> >
> > welcome to the Celix community!
> >
> > 2016-12-16 14:58 GMT+01:00 Roy Lenferink <le...@gmail.com>:
> >
> >> Hi everyone,
> >>
> >> I'm Roy Lenferink, a software engineering student at Saxion University
> of
> >> Applied Sciences.
> >> For my 3rd year I'm having an internship at Thales Nederland.
> >>
> >> Over the past few months I've been working together with Pepijn Noltes,
> >> Erjan Altena and
> >> Gabriele Ricciardi on a (secure) publisher / subscriber implementation,
> >> which is based on
> >> the concepts of the remote service admin (i.e. rsa / topology /
> discovery
> >> pattern).
> >>
> >> Publishers are senders of data, subscribers can receive data. Publishers
> >> can publish/send
> >> data to certain channels (called 'topics' further on), subscribers can
> >> subscribe to these
> >> topics. For every topic a publisher service is created by the pubsub
> >> admin.
> >> This publisher
> >> is announced through etcd. So etcd is used for discovery of the
> >> publishers.
> >> Subscribers are
> >> also registered as a service by the pubsub admin and will watch etcd for
> >> changes and when
> >> a new publisher is announced, the subscriber will check if the topic
> >> matches its interests.
> >> If the subscriber is interested in/subscribed to a certain topic, a
> >> connection between
> >> publisher and subscriber will be instantiated by the pubsub admin.
> >>
> >> The dfi library is used for message serialization. The publisher /
> >> subscriber implementation
> >> will arrange that every message which will be send gets an unique id.
> >>
> >> For communication between publishers and subscribers UDP and ZeroMQ can
> be
> >> used. When using
> >> ZeroMQ it's also possible to setup a secure connection to encrypt the
> >> traffic being send
> >> between publishers and subscribers. This connection can be secured with
> >> ZeroMQ by using
> >> a curve25519 key pair per topic.
> >>
>

It is good to note that the intention of the donation will be that ZMQ is
an optional option and not needed to get publish/subscribe up and running.
The UDP multicast implementation will be the default implementation. This
is needed because ZMQ is LGPL and result is only allowed as optional
option, see [1]

[1] https://www.apache.org/legal/resolved.html#optional

Also for who is interested in the design, Jeroen Nieuwenhuis (a previous
student of Thales I supervised) has implemented a Java version of this
concept and written (IMO) an easy to read report about it [2]. Note that
there are some differences between the C and Java version, but conceptual
they have the same design.

[2]
https://github.com/INAETICS/pub-sub-admin-java/blob/master/inaetics%20pubsub%20design.pdf



> >> The publisher/subscriber implementation supports sending of a single
> >> message and sending
> >> of multipart messages.
> >>
> >> Soon I will create a JIRA issue together with a pull request to get this
> >> part included within Celix.
> >>
> >> If you have any further questions, just reply to this mail!
> >>
> >> Roy
> >>
> >
>

Greetings,
Pepijn

Re: Publisher / subscriber implementation for Celix

[Gabriele Ricciardi <le...@gmail.com>]

Thanks for your contribution, we look forward to integrate your pull
request! :-)

Gabriele

2016-12-19 11:58 GMT+01:00 Gabriele Ricciardi <le...@gmail.com>:

> Hi Roy,
>
> welcome to the Celix community!
>
> 2016-12-16 14:58 GMT+01:00 Roy Lenferink <le...@gmail.com>:
>
>> Hi everyone,
>>
>> I'm Roy Lenferink, a software engineering student at Saxion University of
>> Applied Sciences.
>> For my 3rd year I'm having an internship at Thales Nederland.
>>
>> Over the past few months I've been working together with Pepijn Noltes,
>> Erjan Altena and
>> Gabriele Ricciardi on a (secure) publisher / subscriber implementation,
>> which is based on
>> the concepts of the remote service admin (i.e. rsa / topology / discovery
>> pattern).
>>
>> Publishers are senders of data, subscribers can receive data. Publishers
>> can publish/send
>> data to certain channels (called 'topics' further on), subscribers can
>> subscribe to these
>> topics. For every topic a publisher service is created by the pubsub
>> admin.
>> This publisher
>> is announced through etcd. So etcd is used for discovery of the
>> publishers.
>> Subscribers are
>> also registered as a service by the pubsub admin and will watch etcd for
>> changes and when
>> a new publisher is announced, the subscriber will check if the topic
>> matches its interests.
>> If the subscriber is interested in/subscribed to a certain topic, a
>> connection between
>> publisher and subscriber will be instantiated by the pubsub admin.
>>
>> The dfi library is used for message serialization. The publisher /
>> subscriber implementation
>> will arrange that every message which will be send gets an unique id.
>>
>> For communication between publishers and subscribers UDP and ZeroMQ can be
>> used. When using
>> ZeroMQ it's also possible to setup a secure connection to encrypt the
>> traffic being send
>> between publishers and subscribers. This connection can be secured with
>> ZeroMQ by using
>> a curve25519 key pair per topic.
>>
>> The publisher/subscriber implementation supports sending of a single
>> message and sending
>> of multipart messages.
>>
>> Soon I will create a JIRA issue together with a pull request to get this
>> part included within Celix.
>>
>> If you have any further questions, just reply to this mail!
>>
>> Roy
>>
>
>

Re: Publisher / subscriber implementation for Celix

[Gabriele Ricciardi <le...@gmail.com>]

Hi Roy,

welcome to the Celix community!

2016-12-16 14:58 GMT+01:00 Roy Lenferink <le...@gmail.com>:

> Hi everyone,
>
> I'm Roy Lenferink, a software engineering student at Saxion University of
> Applied Sciences.
> For my 3rd year I'm having an internship at Thales Nederland.
>
> Over the past few months I've been working together with Pepijn Noltes,
> Erjan Altena and
> Gabriele Ricciardi on a (secure) publisher / subscriber implementation,
> which is based on
> the concepts of the remote service admin (i.e. rsa / topology / discovery
> pattern).
>
> Publishers are senders of data, subscribers can receive data. Publishers
> can publish/send
> data to certain channels (called 'topics' further on), subscribers can
> subscribe to these
> topics. For every topic a publisher service is created by the pubsub admin.
> This publisher
> is announced through etcd. So etcd is used for discovery of the publishers.
> Subscribers are
> also registered as a service by the pubsub admin and will watch etcd for
> changes and when
> a new publisher is announced, the subscriber will check if the topic
> matches its interests.
> If the subscriber is interested in/subscribed to a certain topic, a
> connection between
> publisher and subscriber will be instantiated by the pubsub admin.
>
> The dfi library is used for message serialization. The publisher /
> subscriber implementation
> will arrange that every message which will be send gets an unique id.
>
> For communication between publishers and subscribers UDP and ZeroMQ can be
> used. When using
> ZeroMQ it's also possible to setup a secure connection to encrypt the
> traffic being send
> between publishers and subscribers. This connection can be secured with
> ZeroMQ by using
> a curve25519 key pair per topic.
>
> The publisher/subscriber implementation supports sending of a single
> message and sending
> of multipart messages.
>
> Soon I will create a JIRA issue together with a pull request to get this
> part included within Celix.
>
> If you have any further questions, just reply to this mail!
>
> Roy
>