You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by yann ann <xi...@gmail.com> on 2022/10/17 13:05:11 UTC
Re: [Discuss] [Feature] DS can support task running on remote host, not just worker server.
Dear All,
Relate issue: #11652
<https://github.com/apache/dolphinscheduler/issues/11652>
According to the conclusion of the discussion at the regular meeting last
week (2022/10/12), the development of this function is approved. And this
solution has updated to this issue.
- Add table named *t_ds_task_remote_server *in DB.
- Add*TaskRemoteServer *entity in dolphinscheduler-dao model.
- Add a*task remote server* manage page in Security menu, such as
create, delete, edit and test connect.
- Add atask remote server select input (field: taskRemoteServerCode) in
Shell and Python task form.
- Add*task_remote_server_code* column in
*t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log*
tables.
- Add*taskRemoteServerCode *field in
*TaskInstance, TaskDefintion, TaskDefinitionLog, TaskNode*, and add
taskRemoteServerInfo field in TaskInstance (not a table field).
- Add *taskRemoteServerInfo*(just contain ip, user, password, name)
entity in dolphinscheduler-task-plugin model. And add
taskRemoteServerInfofield in TaskExecutionContext.
- Shell and Python task will check
TaskExecutionContext.getTaskRemoteServerInfo(), if not null, it will scp
the command files and resource files to the task remote server, and send
the start command to remote to exec it. If other task plugin need this
feture, can also check this task server field.
- Use JSch 3rd to ssh and scp (scp resource files and command files to
remote server).
- The task will ssh and scp to the task server as the DS running user,
not tenant.
Answers to questions that you are more concerned about：
*1. How to avoid causing IO overload?*
The IO load will be protected by the following measures:
1. When the IO usage of Worker is high, the current task will not be
executed.
2. Set the total size threshold for single-task transfer files.
3. Control transfer rate.
4. Monitor the file transfer process, if the IO usage is too high, the
transfer process will be terminated and the task will be set to fail.
*2. Why not use a remote jar and let the remote server download the
resource files from the resource center?*
I think this will increase the complexity of the remote server's
environment, because the remote server is not part of the DS cluster. For
example, the remote user need have the permission to download the resource
files, and need install JRE.
If there are no more suggestions and questions, I will start development
and submit relevant PRs asap for everyone to review. Thanks.
B. R.
Yann
On Wed, Aug 31, 2022 at 7:54 PM yann ann <xi...@gmail.com> wrote:
> Dear All,
>
> Sorry for the inaccuracy of the previous description about Issue #11652
> <https://github.com/apache/dolphinscheduler/issues/11652> .
>
> *What is the purpose of this feature？*
> DS can support the task instance running on the remote servers (task
> server), not just worker nodes.
>
> - Users can manage these task servers on the page, such as create,
> edit, delete and test connect.
> - Each task node in DAG can specify the task server that needs to be
> executed.
> - The task server property belongs only to task instances, not
> workflow instances.
>
> *MOP*
> 1. Add *TaskServer *Entity in *dolphinscheduler-dao* model, and create a
> table named *t_ds_task_server *in DB. Add TaskServer API.
> [image: image.png]
> 2. Add a task server manage page in Secerity menu, such as create, delete,
> edit and test connect.
> 3. Add a task server select input (field: taskServerCode) in Shell
> and Python task form.
> 4. Add *taskServerCode *column in*
> t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log* tables.
> And change these APIs.
> 5. Add *taskServerCode *field in *TaskInstance, TaskDefintion,
> TaskDefinitionLog, TaskNode*, and add *taskServerInfo *field in *TaskInstance
> *(not a table field).
> 6. Add *taskServerInfo *(just contain ip, user, password, name) entity in
> *dolphinscheduler-task-plugin* model. And add taskServerInfo field in
> *TaskExecutionContext*.
> 7. Shell and Python task will
> check TaskExecutionContext.getTaskServerInfo(), if not null, it will scp
> the command files and resource files to the task server, and send the start
> command to remote to exec it. If other task plugin need this feture, can
> also check this task server field.
> 8. The task will ssh and scp to the task server as the DS running user,
> not tenant.
>
> Look forward to your comments. If no any comment, I will try to complete
> the test and submit a PR. Thanks
>
> B. R.
> Yann
>
> On Sat, Aug 27, 2022 at 9:31 PM yann ann <xi...@gmail.com> wrote:
>
>> Hi,
>>
>> The current DS only supports tasks running on the Worker servers, but in
>> most real business scenarios, tasks need to be run on the specified server,
>> and the server does not need to deploy the Worker service. Please check the Issue
>> #11652 <https://github.com/apache/dolphinscheduler/issues/11652> .
>> Thanks to @SbloodyS for suggesting me to provide the details of the
>> architecture design.
>>
>> Please check the architecture design. I'm not sure if DS needs to use a
>> fixed design template, so I'll try to be as detailed as possible.
>> This design mainly consists of four parts：
>>
>> *Part 1: Create a host table.*
>> Create a table named t_ds_host in db. Please check the following E-R
>> diagram. T_DS_HOST table The purpose of this table design is to save the
>> host information configured in the UI.
>>
>> [image: image.png]
>>
>> *Part 2: Add host manage page.*
>> Add a *Host Manage* module in the Security Page, such as the
>> *Environment Manage* page. Users can add the host records and test
>> connect it.
>> [image: image.png]
>> *Part 3: Add host manage API*
>> Add host manage API, same as environment manage model.
>> About the connect test function, I think that DS needs to test SSH and
>> SFTP at the same time. Because we need to scp our task script to this host
>> and run shell commands. I plan to use the JSch or SSNJ 3rd.
>>
>> *Part 4: Add host selection*
>> Add host selection on Shell and Python task pages.
>> Then shellTask or pythonTask can check this host param, if host param is
>> not null, the task will scp the command files to the remote server and
>> dispatch the exec command.
>>
>> Why just add to Shell and Python tasks?
>> Because I think each task needs to decide for itself if it needs to be
>> executed on the remote server, I think shell and python should add it.
>>
>> look forward to your comments.
>>
>> B. R.
>> Yann
>>
>>
Re: [Discuss] [Feature] DS can support task running on remote host, not just worker server.
Posted by yann ann <xi...@gmail.com>.
Hi Wenjun,
Thanks for your reminder. And i has update the images in
https://github.com/apache/dolphinscheduler/issues/11652
Thanks.
B. R.
Yann
On Tue, Oct 18, 2022 at 11:00 AM wenjun <we...@apache.org> wrote:
> Hi Yann,
>
> The picture cannot display, could you please upload this to the issue#11652
> or other place and add links?
>
> Thanks,
> Wenjun
>
> On Mon, Oct 17, 2022 at 9:06 PM yann ann <xi...@gmail.com> wrote:
>
> > Dear All,
> >
> > Relate issue: #11652
> > <https://github.com/apache/dolphinscheduler/issues/11652>
> >
> > According to the conclusion of the discussion at the regular meeting last
> > week (2022/10/12), the development of this function is approved. And this
> > solution has updated to this issue.
> >
> > - Add table named *t_ds_task_remote_server *in DB.
> > - Add*TaskRemoteServer *entity in dolphinscheduler-dao model.
> > - Add a*task remote server* manage page in Security menu, such as
> > create, delete, edit and test connect.
> > - Add atask remote server select input (field: taskRemoteServerCode)
> > in Shell and Python task form.
> > - Add*task_remote_server_code* column in
> > *t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log*
> > tables.
> > - Add*taskRemoteServerCode *field in
> > *TaskInstance, TaskDefintion, TaskDefinitionLog, TaskNode*, and add
> > taskRemoteServerInfo field in TaskInstance (not a table field).
> > - Add *taskRemoteServerInfo*(just contain ip, user, password, name)
> > entity in dolphinscheduler-task-plugin model. And add
> > taskRemoteServerInfofield in TaskExecutionContext.
> > - Shell and Python task will check
> > TaskExecutionContext.getTaskRemoteServerInfo(), if not null, it will
> > scp the command files and resource files to the task remote server,
> and
> > send the start command to remote to exec it. If other task plugin
> need this
> > feture, can also check this task server field.
> > - Use JSch 3rd to ssh and scp (scp resource files and command files to
> > remote server).
> > - The task will ssh and scp to the task server as the DS running user,
> > not tenant.
> >
> > Answers to questions that you are more concerned about：
> >
> > *1. How to avoid causing IO overload?*
> > The IO load will be protected by the following measures:
> >
> > 1. When the IO usage of Worker is high, the current task will not be
> > executed.
> > 2. Set the total size threshold for single-task transfer files.
> > 3. Control transfer rate.
> > 4. Monitor the file transfer process, if the IO usage is too high, the
> > transfer process will be terminated and the task will be set to fail.
> >
> > *2. Why not use a remote jar and let the remote server download the
> > resource files from the resource center?*
> > I think this will increase the complexity of the remote server's
> > environment, because the remote server is not part of the DS cluster. For
> > example, the remote user need have the permission to download the
> resource
> > files, and need install JRE.
> >
> >
> > If there are no more suggestions and questions, I will start development
> > and submit relevant PRs asap for everyone to review. Thanks.
> >
> > B. R.
> > Yann
> >
> > On Wed, Aug 31, 2022 at 7:54 PM yann ann <xi...@gmail.com>
> wrote:
> >
> >> Dear All,
> >>
> >> Sorry for the inaccuracy of the previous description about Issue #11652
> >> <https://github.com/apache/dolphinscheduler/issues/11652> .
> >>
> >> *What is the purpose of this feature？*
> >> DS can support the task instance running on the remote servers (task
> >> server), not just worker nodes.
> >>
> >> - Users can manage these task servers on the page, such as create,
> >> edit, delete and test connect.
> >> - Each task node in DAG can specify the task server that needs to be
> >> executed.
> >> - The task server property belongs only to task instances, not
> >> workflow instances.
> >>
> >> *MOP*
> >> 1. Add *TaskServer *Entity in *dolphinscheduler-dao* model, and create a
> >> table named *t_ds_task_server *in DB. Add TaskServer API.
> >> [image: image.png]
> >> 2. Add a task server manage page in Secerity menu, such as create,
> >> delete, edit and test connect.
> >> 3. Add a task server select input (field: taskServerCode) in Shell
> >> and Python task form.
> >> 4. Add *taskServerCode *column in*
> >> t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log*
> >> tables. And change these APIs.
> >> 5. Add *taskServerCode *field in *TaskInstance, TaskDefintion,
> >> TaskDefinitionLog, TaskNode*, and add *taskServerInfo *field in
> *TaskInstance
> >> *(not a table field).
> >> 6. Add *taskServerInfo *(just contain ip, user, password, name) entity
> >> in *dolphinscheduler-task-plugin* model. And add taskServerInfo field in
> >> *TaskExecutionContext*.
> >> 7. Shell and Python task will
> >> check TaskExecutionContext.getTaskServerInfo(), if not null, it will scp
> >> the command files and resource files to the task server, and send the
> start
> >> command to remote to exec it. If other task plugin need this feture,
> can
> >> also check this task server field.
> >> 8. The task will ssh and scp to the task server as the DS running user,
> >> not tenant.
> >>
> >> Look forward to your comments. If no any comment, I will try to complete
> >> the test and submit a PR. Thanks
> >>
> >> B. R.
> >> Yann
> >>
> >> On Sat, Aug 27, 2022 at 9:31 PM yann ann <xi...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> The current DS only supports tasks running on the Worker servers, but
> in
> >>> most real business scenarios, tasks need to be run on the specified
> server,
> >>> and the server does not need to deploy the Worker service. Please
> check the Issue
> >>> #11652 <https://github.com/apache/dolphinscheduler/issues/11652> .
> >>> Thanks to @SbloodyS for suggesting me to provide the details of the
> >>> architecture design.
> >>>
> >>> Please check the architecture design. I'm not sure if DS needs to use a
> >>> fixed design template, so I'll try to be as detailed as possible.
> >>> This design mainly consists of four parts：
> >>>
> >>> *Part 1: Create a host table.*
> >>> Create a table named t_ds_host in db. Please check the following E-R
> >>> diagram. T_DS_HOST table The purpose of this table design is to save
> the
> >>> host information configured in the UI.
> >>>
> >>> [image: image.png]
> >>>
> >>> *Part 2: Add host manage page.*
> >>> Add a *Host Manage* module in the Security Page, such as the
> >>> *Environment Manage* page. Users can add the host records and test
> >>> connect it.
> >>> [image: image.png]
> >>> *Part 3: Add host manage API*
> >>> Add host manage API, same as environment manage model.
> >>> About the connect test function, I think that DS needs to test SSH and
> >>> SFTP at the same time. Because we need to scp our task script to this
> host
> >>> and run shell commands. I plan to use the JSch or SSNJ 3rd.
> >>>
> >>> *Part 4: Add host selection*
> >>> Add host selection on Shell and Python task pages.
> >>> Then shellTask or pythonTask can check this host param, if host param
> is
> >>> not null, the task will scp the command files to the remote server and
> >>> dispatch the exec command.
> >>>
> >>> Why just add to Shell and Python tasks?
> >>> Because I think each task needs to decide for itself if it needs to be
> >>> executed on the remote server, I think shell and python should add it.
> >>>
> >>> look forward to your comments.
> >>>
> >>> B. R.
> >>> Yann
> >>>
> >>>
>
Re: [Discuss] [Feature] DS can support task running on remote host, not just worker server.
Posted by wenjun <we...@apache.org>.
Hi Yann,
The picture cannot display, could you please upload this to the issue#11652
or other place and add links?
Thanks,
Wenjun
On Mon, Oct 17, 2022 at 9:06 PM yann ann <xi...@gmail.com> wrote:
> Dear All,
>
> Relate issue: #11652
> <https://github.com/apache/dolphinscheduler/issues/11652>
>
> According to the conclusion of the discussion at the regular meeting last
> week (2022/10/12), the development of this function is approved. And this
> solution has updated to this issue.
>
> - Add table named *t_ds_task_remote_server *in DB.
> - Add*TaskRemoteServer *entity in dolphinscheduler-dao model.
> - Add a*task remote server* manage page in Security menu, such as
> create, delete, edit and test connect.
> - Add atask remote server select input (field: taskRemoteServerCode)
> in Shell and Python task form.
> - Add*task_remote_server_code* column in
> *t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log*
> tables.
> - Add*taskRemoteServerCode *field in
> *TaskInstance, TaskDefintion, TaskDefinitionLog, TaskNode*, and add
> taskRemoteServerInfo field in TaskInstance (not a table field).
> - Add *taskRemoteServerInfo*(just contain ip, user, password, name)
> entity in dolphinscheduler-task-plugin model. And add
> taskRemoteServerInfofield in TaskExecutionContext.
> - Shell and Python task will check
> TaskExecutionContext.getTaskRemoteServerInfo(), if not null, it will
> scp the command files and resource files to the task remote server, and
> send the start command to remote to exec it. If other task plugin need this
> feture, can also check this task server field.
> - Use JSch 3rd to ssh and scp (scp resource files and command files to
> remote server).
> - The task will ssh and scp to the task server as the DS running user,
> not tenant.
>
> Answers to questions that you are more concerned about：
>
> *1. How to avoid causing IO overload?*
> The IO load will be protected by the following measures:
>
> 1. When the IO usage of Worker is high, the current task will not be
> executed.
> 2. Set the total size threshold for single-task transfer files.
> 3. Control transfer rate.
> 4. Monitor the file transfer process, if the IO usage is too high, the
> transfer process will be terminated and the task will be set to fail.
>
> *2. Why not use a remote jar and let the remote server download the
> resource files from the resource center?*
> I think this will increase the complexity of the remote server's
> environment, because the remote server is not part of the DS cluster. For
> example, the remote user need have the permission to download the resource
> files, and need install JRE.
>
>
> If there are no more suggestions and questions, I will start development
> and submit relevant PRs asap for everyone to review. Thanks.
>
> B. R.
> Yann
>
> On Wed, Aug 31, 2022 at 7:54 PM yann ann <xi...@gmail.com> wrote:
>
>> Dear All,
>>
>> Sorry for the inaccuracy of the previous description about Issue #11652
>> <https://github.com/apache/dolphinscheduler/issues/11652> .
>>
>> *What is the purpose of this feature？*
>> DS can support the task instance running on the remote servers (task
>> server), not just worker nodes.
>>
>> - Users can manage these task servers on the page, such as create,
>> edit, delete and test connect.
>> - Each task node in DAG can specify the task server that needs to be
>> executed.
>> - The task server property belongs only to task instances, not
>> workflow instances.
>>
>> *MOP*
>> 1. Add *TaskServer *Entity in *dolphinscheduler-dao* model, and create a
>> table named *t_ds_task_server *in DB. Add TaskServer API.
>> [image: image.png]
>> 2. Add a task server manage page in Secerity menu, such as create,
>> delete, edit and test connect.
>> 3. Add a task server select input (field: taskServerCode) in Shell
>> and Python task form.
>> 4. Add *taskServerCode *column in*
>> t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log*
>> tables. And change these APIs.
>> 5. Add *taskServerCode *field in *TaskInstance, TaskDefintion,
>> TaskDefinitionLog, TaskNode*, and add *taskServerInfo *field in *TaskInstance
>> *(not a table field).
>> 6. Add *taskServerInfo *(just contain ip, user, password, name) entity
>> in *dolphinscheduler-task-plugin* model. And add taskServerInfo field in
>> *TaskExecutionContext*.
>> 7. Shell and Python task will
>> check TaskExecutionContext.getTaskServerInfo(), if not null, it will scp
>> the command files and resource files to the task server, and send the start
>> command to remote to exec it. If other task plugin need this feture, can
>> also check this task server field.
>> 8. The task will ssh and scp to the task server as the DS running user,
>> not tenant.
>>
>> Look forward to your comments. If no any comment, I will try to complete
>> the test and submit a PR. Thanks
>>
>> B. R.
>> Yann
>>
>> On Sat, Aug 27, 2022 at 9:31 PM yann ann <xi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The current DS only supports tasks running on the Worker servers, but in
>>> most real business scenarios, tasks need to be run on the specified server,
>>> and the server does not need to deploy the Worker service. Please check the Issue
>>> #11652 <https://github.com/apache/dolphinscheduler/issues/11652> .
>>> Thanks to @SbloodyS for suggesting me to provide the details of the
>>> architecture design.
>>>
>>> Please check the architecture design. I'm not sure if DS needs to use a
>>> fixed design template, so I'll try to be as detailed as possible.
>>> This design mainly consists of four parts：
>>>
>>> *Part 1: Create a host table.*
>>> Create a table named t_ds_host in db. Please check the following E-R
>>> diagram. T_DS_HOST table The purpose of this table design is to save the
>>> host information configured in the UI.
>>>
>>> [image: image.png]
>>>
>>> *Part 2: Add host manage page.*
>>> Add a *Host Manage* module in the Security Page, such as the
>>> *Environment Manage* page. Users can add the host records and test
>>> connect it.
>>> [image: image.png]
>>> *Part 3: Add host manage API*
>>> Add host manage API, same as environment manage model.
>>> About the connect test function, I think that DS needs to test SSH and
>>> SFTP at the same time. Because we need to scp our task script to this host
>>> and run shell commands. I plan to use the JSch or SSNJ 3rd.
>>>
>>> *Part 4: Add host selection*
>>> Add host selection on Shell and Python task pages.
>>> Then shellTask or pythonTask can check this host param, if host param is
>>> not null, the task will scp the command files to the remote server and
>>> dispatch the exec command.
>>>
>>> Why just add to Shell and Python tasks?
>>> Because I think each task needs to decide for itself if it needs to be
>>> executed on the remote server, I think shell and python should add it.
>>>
>>> look forward to your comments.
>>>
>>> B. R.
>>> Yann
>>>
>>>