You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@iotdb.apache.org by Xiangdong Huang <sa...@gmail.com> on 2021/08/27 06:32:56 UTC

Welcome to join working on "cluster-" branch together

Hi,

I have worked on "cluster-" branch for several days, but still not finished.
As I have no much time to finish it, I'd like to ask cooperators
working on it together.
Please let me know if you want to join.

"cluster-" branch is not for fixing any bug in the cluster module,
but just for refactoring the codes to make the structure similar to
the server module.

Some ideas in the branch:

1. split Thrift RPC service and RPC implementation.
In the server module, we have extracted a class called ThriftService,
which is responsible for starting a RPC service. We just need to
ingest a RPC implementation to it.
However, in the cluster module, some RPC classes mixed them, which
causes code duplication, code inconsistencies, and hard to understand.

2. weaken the role of MetaGroupMember. metaGroupMember is just an
engine for serving meta raft group, which should not control the whole
server too deep. So, many fields (like coordinator, etc.) are
extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB
is responsible for creating them. MetaGroupMember only can modify
these fields, rather than creating them.

That is, the lifecycle of these fields belongs to ClusterIoTDB, rather
than MetaGroupMember.

BTW, MetaGroupMember will be renamed to MetaGroupEngine.

3. Similar with the relationship between StorageEngine and
StorageProcessor in the server module, DataGroupMember can be
considered as StorageProcessor, and we need a DataGroupEngine to
control them.

4. I am considering how to refactor the ClientPool. there are too many
in the codes. But I have no clear idea now.

The 2nd and 3rd refactors will lead to many unexpected issues as I am
not also very familiar with the codes. So, I have to check all UT and
ITs one by one.

I think after the refactoring, developers who know the server module
will be easy to understand the codes in the cluster module.

And, once the ClusterIoTDB dominates the module, it will be clear to
modify the startup process like using ID to replace Node.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Re: Welcome to join working on "cluster-" branch together

Posted by Xiangdong Huang <sa...@gmail.com>.

Hi,

I just finished some works in hand and can find some time back to the
development.

Are there some updates for the branch?
I do not see any PR for this branch. If there is, please let me know
and submit PRs.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Xiangdong Huang <sa...@gmail.com> 于2021年8月31日周二 上午9:12写道：
>
> Hi,
>
> I'd like to open an online discussion at 10:30 AM (Beijing Time) for
> this branch development.
> Welcome to join if you are interested.
> The meeting address and meeting minutes can be found here [1]
>
> [1] https://cwiki.apache.org/confluence/display/IOTDB/Cluster+branch+discussion
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
> Xiangdong Huang <sa...@gmail.com> 于2021年8月27日周五 下午2:32写道：
>
> >
> > Hi,
> >
> > I have worked on "cluster-" branch for several days, but still not finished.
> > As I have no much time to finish it, I'd like to ask cooperators
> > working on it together.
> > Please let me know if you want to join.
> >
> > "cluster-" branch is not for fixing any bug in the cluster module,
> > but just for refactoring the codes to make the structure similar to
> > the server module.
> >
> > Some ideas in the branch:
> >
> > 1. split Thrift RPC service and RPC implementation.
> > In the server module, we have extracted a class called ThriftService,
> > which is responsible for starting a RPC service. We just need to
> > ingest a RPC implementation to it.
> > However, in the cluster module, some RPC classes mixed them, which
> > causes code duplication, code inconsistencies, and hard to understand.
> >
> > 2. weaken the role of MetaGroupMember. metaGroupMember is just an
> > engine for serving meta raft group, which should not control the whole
> > server too deep. So, many fields (like coordinator, etc.) are
> > extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB
> > is responsible for creating them. MetaGroupMember only can modify
> > these fields, rather than creating them.
> >
> > That is, the lifecycle of these fields belongs to ClusterIoTDB, rather
> > than MetaGroupMember.
> >
> > BTW, MetaGroupMember will be renamed to MetaGroupEngine.
> >
> > 3. Similar with the relationship between StorageEngine and
> > StorageProcessor in the server module, DataGroupMember can be
> > considered as StorageProcessor, and we need a DataGroupEngine to
> > control them.
> >
> > 4. I am considering how to refactor the ClientPool. there are too many
> > in the codes. But I have no clear idea now.
> >
> > The 2nd and 3rd refactors will lead to many unexpected issues as I am
> > not also very familiar with the codes. So, I have to check all UT and
> > ITs one by one.
> >
> > I think after the refactoring, developers who know the server module
> > will be easy to understand the codes in the cluster module.
> >
> > And, once the ClusterIoTDB dominates the module, it will be clear to
> > modify the startup process like using ID to replace Node.
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院

Re: Welcome to join working on "cluster-" branch together

Posted by Xiangdong Huang <sa...@gmail.com>.

Hi,

I'd like to open an online discussion at 10:30 AM (Beijing Time) for
this branch development.
Welcome to join if you are interested.
The meeting address and meeting minutes can be found here [1]

[1] https://cwiki.apache.org/confluence/display/IOTDB/Cluster+branch+discussion

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Xiangdong Huang <sa...@gmail.com> 于2021年8月27日周五 下午2:32写道：

>
> Hi,
>
> I have worked on "cluster-" branch for several days, but still not finished.
> As I have no much time to finish it, I'd like to ask cooperators
> working on it together.
> Please let me know if you want to join.
>
> "cluster-" branch is not for fixing any bug in the cluster module,
> but just for refactoring the codes to make the structure similar to
> the server module.
>
> Some ideas in the branch:
>
> 1. split Thrift RPC service and RPC implementation.
> In the server module, we have extracted a class called ThriftService,
> which is responsible for starting a RPC service. We just need to
> ingest a RPC implementation to it.
> However, in the cluster module, some RPC classes mixed them, which
> causes code duplication, code inconsistencies, and hard to understand.
>
> 2. weaken the role of MetaGroupMember. metaGroupMember is just an
> engine for serving meta raft group, which should not control the whole
> server too deep. So, many fields (like coordinator, etc.) are
> extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB
> is responsible for creating them. MetaGroupMember only can modify
> these fields, rather than creating them.
>
> That is, the lifecycle of these fields belongs to ClusterIoTDB, rather
> than MetaGroupMember.
>
> BTW, MetaGroupMember will be renamed to MetaGroupEngine.
>
> 3. Similar with the relationship between StorageEngine and
> StorageProcessor in the server module, DataGroupMember can be
> considered as StorageProcessor, and we need a DataGroupEngine to
> control them.
>
> 4. I am considering how to refactor the ClientPool. there are too many
> in the codes. But I have no clear idea now.
>
> The 2nd and 3rd refactors will lead to many unexpected issues as I am
> not also very familiar with the codes. So, I have to check all UT and
> ITs one by one.
>
> I think after the refactoring, developers who know the server module
> will be easy to understand the codes in the cluster module.
>
> And, once the ClusterIoTDB dominates the module, it will be clear to
> modify the startup process like using ID to replace Node.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院

答复: Re: Welcome to join working on "cluster-" branch together

Posted by 程建云 <ch...@360.cn>.

Sorry for the inconvenience for the email title change. Later I'll use my personal email account(chengjianyun@outlook.com) to avoid the issue.


Actually, I don't where I could start from. When I checked out the `cluster-` branch, I saw so many changes... Could we book a time to talk about the form of cooperation and how community member get start? Anyone in the community can join and I can take the meeting notes and publish to community after the meeting.


Thanks!

Jianyun Cheng

________________________________
发件人: Xiangdong Huang <sa...@gmail.com>
发送时间: 2021年8月28日 10:49:04
收件人: dev
主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Re: Welcome to join working on "cluster-" branch together

Hi,

It is ok. online meeting is fine, but we need to leave a meeting
records to the community.

Before that, I noticed that the email is marked as "【可能伪造邮件 POSSIBLY SPOOFED 】".

Please change some settings of your mailbox... All these will be
recorded on the Internet and it is not good for searching content...

(BTW, if you want to reply on this thread, please modify the subject manually)

Best
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

程建云 <ch...@360.cn> 于2021年8月27日周五 下午6:37写道：
>
> Hi, Xiangdong
>
>
> We are glad to join in to work together to help refactor the cluster module!
>
>
> Do you have any idea about the form of cooperation? It's better to have some regular online meeting or something like to help improve the productivity.
>
>
>
> Thanks!
>
> Jianyun Cheng
>
> ________________________________
> 发件人: 李思佳 <li...@360.cn>
> 发送时间: 2021年8月27日 18:25
> 收件人: dev@iotdb.apache.org
> 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
>
> Hi,
>
> This is Sijia Li, I am a big data engineer of Qihoo 360.
>
> I'd like to participate in this work with my colleague @Jianyun Cheng.
>
> I hope that we can have a pleasant cooperation.
>
> Thanks
>
> -----邮件原件-----
> 发件人: Xiangdong Huang <sa...@gmail.com>
> 发送时间: 2021年8月27日 14:33
> 收件人: dev <de...@iotdb.apache.org>
> 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
>
> Hi,
>
> I have worked on "cluster-" branch for several days, but still not finished.
> As I have no much time to finish it, I'd like to ask cooperators working on it together.
> Please let me know if you want to join.
>
> "cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module.
>
> Some ideas in the branch:
>
> 1. split Thrift RPC service and RPC implementation.
> In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it.
> However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand.
>
> 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them.
>
> That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember.
>
> BTW, MetaGroupMember will be renamed to MetaGroupEngine.
>
> 3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them.
>
> 4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now.
>
> The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one.
>
> I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module.
>
> And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院

Re: Welcome to join working on "cluster-" branch together

Posted by Xiangdong Huang <sa...@gmail.com>.

Hi,

It is ok. online meeting is fine, but we need to leave a meeting
records to the community.

Before that, I noticed that the email is marked as "【可能伪造邮件 POSSIBLY SPOOFED 】".

Please change some settings of your mailbox... All these will be
recorded on the Internet and it is not good for searching content...

(BTW, if you want to reply on this thread, please modify the subject manually)

Best
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

程建云 <ch...@360.cn> 于2021年8月27日周五 下午6:37写道：
>
> Hi, Xiangdong
>
>
> We are glad to join in to work together to help refactor the cluster module!
>
>
> Do you have any idea about the form of cooperation? It's better to have some regular online meeting or something like to help improve the productivity.
>
>
>
> Thanks!
>
> Jianyun Cheng
>
> ________________________________
> 发件人: 李思佳 <li...@360.cn>
> 发送时间: 2021年8月27日 18:25
> 收件人: dev@iotdb.apache.org
> 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
>
> Hi,
>
> This is Sijia Li, I am a big data engineer of Qihoo 360.
>
> I'd like to participate in this work with my colleague @Jianyun Cheng.
>
> I hope that we can have a pleasant cooperation.
>
> Thanks
>
> -----邮件原件-----
> 发件人: Xiangdong Huang <sa...@gmail.com>
> 发送时间: 2021年8月27日 14:33
> 收件人: dev <de...@iotdb.apache.org>
> 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
>
> Hi,
>
> I have worked on "cluster-" branch for several days, but still not finished.
> As I have no much time to finish it, I'd like to ask cooperators working on it together.
> Please let me know if you want to join.
>
> "cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module.
>
> Some ideas in the branch:
>
> 1. split Thrift RPC service and RPC implementation.
> In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it.
> However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand.
>
> 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them.
>
> That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember.
>
> BTW, MetaGroupMember will be renamed to MetaGroupEngine.
>
> 3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them.
>
> 4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now.
>
> The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one.
>
> I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module.
>
> And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院

答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together

Posted by 程建云 <ch...@360.cn>.

Hi, Xiangdong

We are glad to join in to work together to help refactor the cluster module!

Do you have any idea about the form of cooperation? It's better to have some regular online meeting or something like to help improve the productivity.

Thanks!

Jianyun Cheng

________________________________
发件人: 李思佳 <li...@360.cn>
发送时间: 2021年8月27日 18:25
收件人: dev@iotdb.apache.org
主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together

Hi,

This is Sijia Li, I am a big data engineer of Qihoo 360.

I'd like to participate in this work with my colleague @Jianyun Cheng.

I hope that we can have a pleasant cooperation.

Thanks

-----邮件原件-----
发件人: Xiangdong Huang <sa...@gmail.com>
发送时间: 2021年8月27日 14:33
收件人: dev <de...@iotdb.apache.org>
主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together

Hi,

I have worked on "cluster-" branch for several days, but still not finished.
As I have no much time to finish it, I'd like to ask cooperators working on it together.
Please let me know if you want to join.

"cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module.

Some ideas in the branch:

1. split Thrift RPC service and RPC implementation.
In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it.
However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand.

2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them.

That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember.

BTW, MetaGroupMember will be renamed to MetaGroupEngine.

3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them.

4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now.

The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one.

I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module.

And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

黄向东
清华大学 软件学院

答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together

Posted by 李思佳 <li...@360.cn>.

Hi,

This is Sijia Li, I am a big data engineer of Qihoo 360.

I'd like to participate in this work with my colleague @Jianyun Cheng.

I hope that we can have a pleasant cooperation.

Thanks

Hi,

"cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module.

Some ideas in the branch:

That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember.

BTW, MetaGroupMember will be renamed to MetaGroupEngine.

3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them.

4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now.

The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one.

I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module.

And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

黄向东
清华大学 软件学院