You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by Xinyu Tan <tx...@gmail.com> on 2023/02/21 16:45:04 UTC
【Thanks from the Apache IoTDB community and future plans with Apache Ratis】
Hi, Ratis CommunityRatis Community Community
I'm Xinyu Tan from the Apache IoTDB community. My Github ID is
OneSizeFitsQuorum, and I'm interested in consensus algorithm, distributed
storage system, time-series database and distributed transaction.
In the past nearly a year, the consensus layer of Apache IoTDB has
integrated with Apache Ratis, which makes Apache IoTDB capable of strong
consistency. Many thanks to Apache Ratis community especially Tsz Wo Sze
for your patient help and the efforts of William.
In the next year, apart from William, another student and I will also
devote our energy to the stability, performance, memory control, snapshot
transmission and other issues of Apache Ratis, hoping to make Apache Ratis
and Apache IoTDB better and better.
Finally, I would like to know if there is any overall roadmap for Apache
Ratis in 2023 or release/3.0, so that we can review some of the issues that
Apache IoTDB is having with Apache Ratis to do something more beneficial
for both of us.
Thanks
----------------------
Xinyu Tan
Re: 【Thanks from the Apache IoTDB community and future plans with Apache Ratis】
Posted by Kaijie Chen <ck...@apache.org>.
Thanks Xinyu for bring up this discussion.
I'm from the Apache Ozone community and Alluxio community,
and I'm also interested in improving Apache Ratis.
Let's work together to make it better.
Best,
Kaijie Chen
On 2023/02/21 16:45:04 Xinyu Tan wrote:
> Hi, Ratis CommunityRatis Community Community
>
> I'm Xinyu Tan from the Apache IoTDB community. My Github ID is
> OneSizeFitsQuorum, and I'm interested in consensus algorithm, distributed
> storage system, time-series database and distributed transaction.
>
> In the past nearly a year, the consensus layer of Apache IoTDB has
> integrated with Apache Ratis, which makes Apache IoTDB capable of strong
> consistency. Many thanks to Apache Ratis community especially Tsz Wo Sze
> for your patient help and the efforts of William.
>
> In the next year, apart from William, another student and I will also
> devote our energy to the stability, performance, memory control, snapshot
> transmission and other issues of Apache Ratis, hoping to make Apache Ratis
> and Apache IoTDB better and better.
>
> Finally, I would like to know if there is any overall roadmap for Apache
> Ratis in 2023 or release/3.0, so that we can review some of the issues that
> Apache IoTDB is having with Apache Ratis to do something more beneficial
> for both of us.
>
> Thanks
> ----------------------
> Xinyu Tan
>
Re: 【Thanks from the Apache IoTDB community and future plans with Apache Ratis】
Posted by Xinyu Tan <tx...@gmail.com>.
Thanks Kaijie and Tsz-Wo~
Let's work together to make Apache Ratis better and better.
Best
-------------------------
Xinyu Tan
Tsz Wo Sze <sz...@gmail.com> 于2023年2月22日周三 13:58写道：
> Hi Xinyu,
>
> Welcome to Apache Ratis. Thanks a lot for joining us working on it!
>
> - One idea is to use Streaming to do appendEntries. gRPC seems not
> suitable to do appendEntries when the data size is large since it requires
> buffer allocation and buffer copying. It is better to use Netty zero
> buffer copying, i.e. Streaming.
>
> - For the data stream feature, there is a missing component -- Currently,
> it requires all nodes to be available during streaming. AppendEntries
> won't include streamed data. So if a follower does not have the stream
> data, it must install a snapshot in order to catch up. In Ozone Write
> Pipeline, we do have an assumption that all datanodes are available in the
> pipeline. When one of the datanodes fails, it closes the pipeline (so that
> it does not have to take care of resending streamed data). Since you
> seem to be interested in the data stream feature, you may consider working
> on this missing component.
>
> - Another idea is to support reading using Streaming. Then, reading can
> also have Netty zero buffer copying.
>
> These are my wish list items.
>
> Tsz-Wo
>
> On Tue, Feb 21, 2023 at 8:45 AM Xinyu Tan <tx...@gmail.com> wrote:
>
> > Hi, Ratis CommunityRatis Community Community
> >
> > I'm Xinyu Tan from the Apache IoTDB community. My Github ID is
> > OneSizeFitsQuorum, and I'm interested in consensus algorithm, distributed
> > storage system, time-series database and distributed transaction.
> >
> > In the past nearly a year, the consensus layer of Apache IoTDB has
> > integrated with Apache Ratis, which makes Apache IoTDB capable of strong
> > consistency. Many thanks to Apache Ratis community especially Tsz Wo Sze
> > for your patient help and the efforts of William.
> >
> > In the next year, apart from William, another student and I will also
> > devote our energy to the stability, performance, memory control, snapshot
> > transmission and other issues of Apache Ratis, hoping to make Apache
> Ratis
> > and Apache IoTDB better and better.
> >
> > Finally, I would like to know if there is any overall roadmap for Apache
> > Ratis in 2023 or release/3.0, so that we can review some of the issues
> that
> > Apache IoTDB is having with Apache Ratis to do something more beneficial
> > for both of us.
> >
> > Thanks
> > ----------------------
> > Xinyu Tan
> >
>
Re: 【Thanks from the Apache IoTDB community and future plans with Apache Ratis】
Posted by Tsz Wo Sze <sz...@gmail.com>.
Hi Xinyu,
Welcome to Apache Ratis. Thanks a lot for joining us working on it!
- One idea is to use Streaming to do appendEntries. gRPC seems not
suitable to do appendEntries when the data size is large since it requires
buffer allocation and buffer copying. It is better to use Netty zero
buffer copying, i.e. Streaming.
- For the data stream feature, there is a missing component -- Currently,
it requires all nodes to be available during streaming. AppendEntries
won't include streamed data. So if a follower does not have the stream
data, it must install a snapshot in order to catch up. In Ozone Write
Pipeline, we do have an assumption that all datanodes are available in the
pipeline. When one of the datanodes fails, it closes the pipeline (so that
it does not have to take care of resending streamed data). Since you
seem to be interested in the data stream feature, you may consider working
on this missing component.
- Another idea is to support reading using Streaming. Then, reading can
also have Netty zero buffer copying.
These are my wish list items.
Tsz-Wo
On Tue, Feb 21, 2023 at 8:45 AM Xinyu Tan <tx...@gmail.com> wrote:
> Hi, Ratis CommunityRatis Community Community
>
> I'm Xinyu Tan from the Apache IoTDB community. My Github ID is
> OneSizeFitsQuorum, and I'm interested in consensus algorithm, distributed
> storage system, time-series database and distributed transaction.
>
> In the past nearly a year, the consensus layer of Apache IoTDB has
> integrated with Apache Ratis, which makes Apache IoTDB capable of strong
> consistency. Many thanks to Apache Ratis community especially Tsz Wo Sze
> for your patient help and the efforts of William.
>
> In the next year, apart from William, another student and I will also
> devote our energy to the stability, performance, memory control, snapshot
> transmission and other issues of Apache Ratis, hoping to make Apache Ratis
> and Apache IoTDB better and better.
>
> Finally, I would like to know if there is any overall roadmap for Apache
> Ratis in 2023 or release/3.0, so that we can review some of the issues that
> Apache IoTDB is having with Apache Ratis to do something more beneficial
> for both of us.
>
> Thanks
> ----------------------
> Xinyu Tan
>