You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by gaojun2048 <ga...@apache.org> on 2024/04/10 02:23:53 UTC
[Proposal] STIP7-The Zeta Master is separated from the Worker
At present, the Zeta engine does not distinguish between master nodes and
worker nodes. All nodes can serve as master and worker nodes, and only one
node in the entire cluster is marked as a master node at the same time. The
master is responsible for receiving tasks submitted by clients and
scheduling tasks, while also taking on the task running work.
When there are many tasks running in the cluster, the load on the master
node can be very high, which may cause the master node to hang up and
perform fault tolerance on the master node. Master node fault tolerance
will further increase the responsibility of the cluster, especially newly
selected master nodes have a greater chance of failure due to excessive
load.
The goal of this proposal is to separate the roles of the master and worker
nodes. The master node is only responsible for task scheduling, IMAP data
storage, and access; Worker nodes are only responsible for the operation of
tasks, and do not participate in the storage and access of IMAP data, nor
do they participate in parameter elections.
Detailed design document link：
https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP7-The+Zeta+Master+is+separated+from+the+Worker
--
Best Regards
------------
Apache ID: gaojun2048
Github ID: EricJoy2048
Mail: gaojun2048@gmail.com
Re: [Proposal] STIP7-The Zeta Master is separated from the Worker
Posted by Jia Fan <fa...@qq.com.INVALID>.
+1. The new deployment mode can make clusters more stable
________________________
Jia Fan
> 2024年4月13日 21:42，hailin0 <wa...@apache.org> 写道：
>
> good idea!
>
> Need to clarify how to test new cluster architecture
>
> gaojun2048 <ga...@apache.org> 于2024年4月10日周三 10:24写道：
>
>> At present, the Zeta engine does not distinguish between master nodes and
>> worker nodes. All nodes can serve as master and worker nodes, and only one
>> node in the entire cluster is marked as a master node at the same time. The
>> master is responsible for receiving tasks submitted by clients and
>> scheduling tasks, while also taking on the task running work.
>>
>> When there are many tasks running in the cluster, the load on the master
>> node can be very high, which may cause the master node to hang up and
>> perform fault tolerance on the master node. Master node fault tolerance
>> will further increase the responsibility of the cluster, especially newly
>> selected master nodes have a greater chance of failure due to excessive
>> load.
>>
>> The goal of this proposal is to separate the roles of the master and worker
>> nodes. The master node is only responsible for task scheduling, IMAP data
>> storage, and access; Worker nodes are only responsible for the operation of
>> tasks, and do not participate in the storage and access of IMAP data, nor
>> do they participate in parameter elections.
>>
>>
>> Detailed design document link：
>>
>> https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP7-The+Zeta+Master+is+separated+from+the+Worker
>>
>> --
>>
>> Best Regards
>>
>> ------------
>>
>> Apache ID: gaojun2048
>>
>> Github ID: EricJoy2048
>>
>> Mail: gaojun2048@gmail.com
>>
Re: [Proposal] STIP7-The Zeta Master is separated from the Worker
Posted by hailin0 <wa...@apache.org>.
good idea!
Need to clarify how to test new cluster architecture
gaojun2048 <ga...@apache.org> 于2024年4月10日周三 10:24写道：
> At present, the Zeta engine does not distinguish between master nodes and
> worker nodes. All nodes can serve as master and worker nodes, and only one
> node in the entire cluster is marked as a master node at the same time. The
> master is responsible for receiving tasks submitted by clients and
> scheduling tasks, while also taking on the task running work.
>
> When there are many tasks running in the cluster, the load on the master
> node can be very high, which may cause the master node to hang up and
> perform fault tolerance on the master node. Master node fault tolerance
> will further increase the responsibility of the cluster, especially newly
> selected master nodes have a greater chance of failure due to excessive
> load.
>
> The goal of this proposal is to separate the roles of the master and worker
> nodes. The master node is only responsible for task scheduling, IMAP data
> storage, and access; Worker nodes are only responsible for the operation of
> tasks, and do not participate in the storage and access of IMAP data, nor
> do they participate in parameter elections.
>
>
> Detailed design document link：
>
> https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP7-The+Zeta+Master+is+separated+from+the+Worker
>
> --
>
> Best Regards
>
> ------------
>
> Apache ID: gaojun2048
>
> Github ID: EricJoy2048
>
> Mail: gaojun2048@gmail.com
>