Custom message serialization

[Alexey Goncharuk <ag...@apache.org>]

Hello Ratis community, a fellow Apache member here :)

I am currently researching an option to integrate a consensus module for
Apache Ignite, mostly as metadata storage and group membership service in
the beginning. I could not help noticing that there is a neighbor Apache
project that looks very promising.

I am just getting started and see that the messaging layer is separated
from the core Raft algorithm implementation, but the message definitions
and serialization are Google Protobuf-generated. In Ignite, we have a
custom serialization protocol and try to keep the core module as small
as possible, so the protobuf dependency does not quite fit our project.

I was wondering if you have any plans for abstracting the serialization
layer as well, or perhaps would accept a contribution that implements such
a separation (after the design is discussed here, of course)? Or, perhaps,
I missed something and I can plug custom messages to the Ratis core somehow?

Thanks in advance,
--Alexey Goncharuk

Re: Custom message serialization

[Tsz Wo Sze <sz...@gmail.com>]

Hi Alexey,

I believe we are open to custom message serialization, especially,
Ratis is designed to be highly pluggable.  Indeed, it is a good time
to do so since we still do not yet need worry about backward
compatibility, since Ratis is still not yet GA.  Recently we do see
some weakness in Protobuf such as unnecessary buffer copying in some
cases.

It would be great if you could contribute in Ratis!  Contributions are
always welcome.  :)

Thanks for reaching out.
Tsz-Wo


On Fri, Oct 4, 2019 at 2:10 AM Alexey Goncharuk <ag...@apache.org> wrote:
>
> Hello Ratis community, a fellow Apache member here :)
>
> I am currently researching an option to integrate a consensus module for
> Apache Ignite, mostly as metadata storage and group membership service in
> the beginning. I could not help noticing that there is a neighbor Apache
> project that looks very promising.
>
> I am just getting started and see that the messaging layer is separated
> from the core Raft algorithm implementation, but the message definitions
> and serialization are Google Protobuf-generated. In Ignite, we have a
> custom serialization protocol and try to keep the core module as small
> as possible, so the protobuf dependency does not quite fit our project.
>
> I was wondering if you have any plans for abstracting the serialization
> layer as well, or perhaps would accept a contribution that implements such
> a separation (after the design is discussed here, of course)? Or, perhaps,
> I missed something and I can plug custom messages to the Ratis core somehow?
>
> Thanks in advance,
> --Alexey Goncharuk