You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by lidong dai <da...@gmail.com> on 2020/05/20 11:12:57 UTC
Re: [Feature] Support SSH Task
glad to hear that you will implement this feature
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <pe...@qq.com> 于2020年5月20日周三 下午3:47写道：
> My code is not perfect yet. I will write a detailed design document. Then
> I will realize this feature about our discussion result.
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "wenhemin"<whm_777@163.com>;
> 发送时间: 2020年5月18日(星期一) 晚上7:50
> 收件人: "裴龙武"<peilongwu@qq.com>;"dev"<dev@dolphinscheduler.apache.org
> >;
>
> 主题: Re: [Feature] Support SSH Task and Support dummy task like airflow
>
>
>
> Thanks for writing detailed documentation. I think this is also a missing
> feature of DS.
> About the extension point:
> 1.Can ssh tasks be merged into shell tasks. Essentially, they all execute
> shell commands.
> 2.About dummy task, DS has the function of disable nodes, I do n’t know if
> this requirement is met.
>
> The script from AirFlow to Dolphin is great.
>
> > 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com> 写道：
> >
> >
> > OK, 3Q!
> >
> > First, I will ensure that open source can use.
> >
> > Second, I think we must discuss deeply. I write a more detailed
> document. You can check the attachment. I also send the document to
> DaiLidong.
> >
> > Third, I'll give you the error of not using SSH connection pool.
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "wenhemin"<whm_777@163.com>;
> > 发送时间: 2020年5月14日(星期四) 晚上7:26
> > 收件人: "裴龙武"<peilongwu@qq.com>;
> > 主题: Re: [Feature] Support SSH Task and Support dummy task like airflow
> >
> > Great!
> > I think, Can ssh tasks be merged into shell tasks, execute
> script locally or remotely, Configure on the front end.
> > About ssh connect pool, I did not find it necessary to use the
> connection pool.
> >
> > BTW, Look at the code to introduce additional jar packages, You also
> need to ensure that open source can use the license of this jar package.
> >
> >> 在 2020年5月14日，16:20，裴龙武 <peilongwu@qq.com <mailto:peilongwu@qq.com>>
> 写道：
> >>
> >>
> >> 1. The priority between these tasks is also depended on the
> dolphin DAG define. When the front task is not finished, it not execute
> next task.
> >>
> >> 2. I extend ssh task. I also use local params to config ssh host,
> user and password.
> >>
> >> E.g:
> >> public static AbstractTask newTask(TaskExecutionContext
> taskExecutionContext, Logger logger)
> >> throws IllegalArgumentException {
> >> Boolean enable =
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> >> if (enable != null && enable == false ) {
> >> return new
> DummyTask(taskExecutionContext, logger);
> >> }
> >> switch
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> >> case SHELL:
> >> return new
> ShellTask(taskExecutionContext, logger);
> >> case PROCEDURE:
> >> return new
> ProcedureTask(taskExecutionContext, logger);
> >> case SQL:
> >> return new
> SqlTask(taskExecutionContext, logger);
> >> case MR:
> >> return new
> MapReduceTask(taskExecutionContext, logger);
> >> case SPARK:
> >> return new
> SparkTask(taskExecutionContext, logger);
> >> case FLINK:
> >> return new
> FlinkTask(taskExecutionContext, logger);
> >> case PYTHON:
> >> return new
> PythonTask(taskExecutionContext, logger);
> >> case HTTP:
> >> return new
> HttpTask(taskExecutionContext, logger);
> >> case DATAX:
> >> return new
> DataxTask(taskExecutionContext, logger);
> >> case SQOOP:
> >> return new
> SqoopTask(taskExecutionContext, logger);
> >> case SSH:
> >> return new
> SSHTask(taskExecutionContext, logger);
> >> default:
> >> logger.error("unsupport task
> type: {}", taskExecutionContext.getTaskType());
> >> throw new
> IllegalArgumentException("not support task type");
> >> }
> >> }
> >> 3. I am not sure that it supports window or not.
> >>
> >>
> >>
> >> ------------------ 原始邮件 ------------------
> >> 发件人: "wenhemin"<whm_777@163.com <mailto:whm_777@163.com>>;
> >> 发送时间: 2020年5月14日(星期四) 下午3:46
> >> 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com>>;
> >> 主题: Re: [Feature] Support SSH Task and Support dummy task like
> airflow
> >>
> >> Sorry, My previous description is not very clear.
> >>
> >> I want to ask some questions:
> >> 1.How to control the priority between ssh tasks? There may be
> some ssh tasks that have been waiting for execution.
> >> 2.I understand what you want to solve is the problem of executing
> remote ssh scripts in batches.
> >> So, not sure how to use this function.
> >> 3.I don't know if this supports windows system.
> >>
> >>> 在 2020年5月13日，20:56，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com>> 写道：
> >>>
> >>>
> >>> I use spin lock. Here is my code. Of course , it's not
> perfect. I just do a test. To my surprise, it is the result of the
> execution is the same as the AirFlow
> >>>
> >>> 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> 相同的结果。
> >>>
> >>>
> >>>
> >>>
> >>> ------------------ 原始邮件 ------------------
> >>> 发件人: "whm_777"<whm_777@163.com <mailto:whm_777@163.com
> >>;
> >>> 发送时间: 2020年5月13日(星期三) 晚上7:21
> >>> 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com>>;
> >>> 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> >>>
> >>> You can modify the maximum number of linux ssh connections.
> >>> If use ssh connection pool, How to control the priority of
> ssh?
> >>>
> >>>> 在 2020年5月13日，18:01，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com>> 写道：
> >>>>
> >>>>
> >>>> First 3Q，
> >>>>
> >>>> I use more than 100 task node. But SSH connections
> are limited.
> >>>>
> >>>>
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> >>>> <330BE502@F7F80E73.76C5BB5E.jpg>
> >>>>
> >>>>
> >>>>
> >>>> ------------------ 原始邮件 ------------------
> >>>> 发件人: "whm_777"<whm_777@163.com <mailto:whm_777@163.com
> >>;
> >>>> 发送时间: 2020年5月13日(星期三) 下午5:50
> >>>> 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com
> >>;
> >>>> 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> >>>>
> >>>> E.g.
> >>>> rtn_code=`ssh -o ServerAliveInterval=60 -p xxxx
> root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx> ‘shell
> command >/dev/null 2>&1; echo $?'`
> >>>> if [ "$rtn_code" -eq 0 ]; then
> >>>> echo "成功"
> >>>> exit 0
> >>>> else
> >>>> echo "失败"
> >>>> exit 1
> >>>> fi
> >>>>
> >>>> Batch shell command is not supported.
> >>>> Multiple servers can be split into multiple task nodes.
> >>>>
> >>>>> 在 2020年5月13日，17:40，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com>> 写道：
> >>>>>
> >>>>>
> >>>>> Could you give me a example，3Q. 能否给我一个例子，谢谢！
> >>>>>
> >>>>> By the way, I have more than 100 tasks in one DAG.
> These tasks connect two other server to execute. So SSH tasks must have
> pool to manager. Now I use JSch and realize a simple pool.
> >>>>>
> >>>>> 顺带说一下，在我的实际场景中，我有100多个 SSH 任务，这些任务连接两台任务服务器进行任务执行。所以
> SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> >>>>>
> >>>>> ------------------ 原始邮件 ------------------
> >>>>> 发件人: "wenhemin"<whm_777@163.com <mailto:
> whm_777@163.com>>;
> >>>>> 发送时间: 2020年5月13日(星期三) 下午5:24
> >>>>> 收件人: "dev"<dev@dolphinscheduler.apache.org <mailto:
> dev@dolphinscheduler.apache.org>>;
> >>>>> 主题: Re: [Feature] Support SSH Task and Support dummy
> task like airflow
> >>>>>
> >>>>> The shell node is supports remote calling, and get
> the remote command result code.
> >>>>>
> >>>>>
> >>>>> > 在 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> <mailto:peilongwu@qq.com>> 写道：
> >>>>> >
> >>>>> > Dear ALL：
> >>>>> >
> >>>>> >
> >>>>> > Support Linux SSH Task 支持 Linux SSH 任务
> >>>>> >
> >>>>> > 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> >>>>> >
> >>>>> > For example, in my project, the workflow's tasks
> want to execute shell scripts where are in different server's different
> directory. When worker execute these shell scripts, it must use the same
> user to login these server. Also, the worker can get the executing state of
> these server. We can config these server 's host,user and password.
> >>>>> >
> >>>>> > SSH Task is very useful for most user SSH
> 任务对大多数用户是非常有用的
> >>>>> >
> >>>>> > 分布式调度任务所执行的 Shell
> 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> >>>>> >
> >>>>> > In dolphinscheduler, the most executing tasks
> are in different servers who are not workers. These servers also have their
> different fixed services. We just have to pass different parameters to
> schedule these shell scripts to execute.
> >>>>> >
> >>>>> > Python has a module to execute ssh script Python
> 有固定的工具包，可执行这些SSH Shell 脚本
> >>>>> >
> >>>>> > Python 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> >>>>> >
> >>>>> > Python has a module that can execute SSH Shell
> script. It's paramiko.
> >>>>> >
> >>>>> > Others 其他内容
> >>>>> >
> >>>>> > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> >>>>> >
> >>>>> > I found this described in previous feature, but
> it was relatively simple.
> >>>>> > Feature URL
> >>>>> >
> >>>>> > 另外，我通过 Shell Task
> 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> >>>>> >
> >>>>> > In addition, it is very inconvenient for me to
> perform remote tasks through Shell Task. Here is my script. I don't know if
> there's a better way.
> >>>>> > sshpass -p 'password' ssh user@host echo 'ssh
> success' echo 'Hello World' -&gt; /home/dolphinscheduler/test/hello.txt
> echo 'end'
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > Support dummy task like airflow 支持像 Airflow
> 中的虚拟任务
> >>>>> >
> >>>>> > 场景描述：项目中，有已经产品化的 DAG 文件，DAG
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为 Dummy
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> >>>>> >
> >>>>> > For example, in my project, it has a productized
> DAG file. The file contains different modules, some of which are
> interdependent and some of which are not. When customers purchase different
> modules, we need to set some tasks as dummy tasks, which some modules are
> not purchased and the purchased module is not dependent. Because of this
> setting, these dummy tasks are actually not executed. The benefits of this
> setup are product unity and diagram integrity. In airflow, these task
> execute by dummy operator.
> >>>>> >
> >>>>> > ** Realize 实现方式**
> >>>>> >
> >>>>> > Dummy Task 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> dummy 时，实际的任务不执行，执行 Dummy Task。
> >>>>> >
> >>>>> > Dummy Task is easy to realize, but it need to
> use with other different tasks. When the task's executed type is set to
> dummy type, the task are executed as a dummy task and the real task is not
> executed.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> >>>>> >
> >>>>> > By the way，I already realize these two&nbsp;
> features in my fork branch.&nbsp;Whether the follow-up release can be
> supported
> >>>>>
> >>>>
> >>>
> >>> <SSHClient.java><SSHPool.java><SSHTask.java>
> >>
> >
> > <项目场景中关于Dolphin的一些扩展点.pdf>
Re: [Feature] Support SSH Task
Posted by lidong dai <da...@gmail.com>.
good, if you meet some problems, please leave message
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <pe...@qq.com> 于2020年5月28日周四 上午9:23写道：
> OK, I'll try to do it by the way we talked about it.
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "lidong dai"<dailidong66@gmail.com>;
> 发送时间: 2020年5月27日(星期三) 晚上7:51
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> I think your idea is good, haha
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> Hemin Wen <wenhemin@apache.org> 于2020年5月27日周三 上午11:53写道：
>
> > Is it possible to extend the data source and increase the ssh type.
> > I look at the data structure of DB is ok.
> >
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > 裴龙武 <peilongwu@qq.com> 于2020年5月27日周三 上午10:55写道：
> >
> > > Good idea! It's can solve the exceptions where the numbers of SSH
> > > connections exceeds.
> > >
> > >
> > > We also have another problem of how to config remote server
> easily when
> > we
> > > have hundreds of tasks.
> > >
> > >
> > > Now I use the local param to config remote server's
> IP,username,password.
> > > But every task needs to config.
> > >
> > >
> > >
> > >
> > >
> > >
> > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> > > 发送时间:&nbsp;2020年5月26日(星期二) 晚上6:38
> > > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> > >
> > > 主题:&nbsp;Re: [Feature] Support SSH Task
> > >
> > >
> > >
> > > good idea, I think "Control the number of task's concurrent" may
> be have
> > > some other methods,&nbsp; for now, we don't have this
> feature, but when I
> > > communicate with the team of China Unicom， they provide a new
> solve
> > > solution：
> > > &nbsp; we can add a task concurrent page, you can set a
> variable value
> > like
> > > 'ETL_JOB' =10 and so on, then give the task type add a option for
> > choosing
> > > ETL_JOB or other variable, and there may be many task also
> choose ETL_JOB
> > > variable.&nbsp; when these tasks begin to running at the
> same time, the
> > > Master
> > > Server will check whether a task begin to run or wait(if the
> running
> > tasks
> > > num = 10)
> > >
> > >
> > > Best egards
> > > ---------------
> > > DolphinScheduler(Incubator) PPMC
> > > Lidong Dai 代立冬
> > > dailidong66@gmail.com
> > > ---------------
> > >
> > >
> > > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月26日周二 下午2:34写道：
> > >
> > > &gt; There is only two ways that I can think to do about
> this.
> > > &gt;
> > > &gt;
> > > &gt; 1. Simple SSH connect pool
> > > &gt; 2. Control the number of task's concurrent
> > > &gt;
> > > &gt;
> > > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > > &gt; 发件人:&amp;nbsp;"Hemin Wen"<wenhemin@apache.org
> &amp;gt;;
> > > &gt; 发送时间:&amp;nbsp;2020年5月22日(星期五) 晚上7:29
> > > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;gt;;
> > > &gt;
> > > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt; What other friends think about this discuss
> > > &gt;
> > > &gt;
> > > &gt; --------------------
> > > &gt; DolphinScheduler(Incubator) Commtter
> > > &gt; Hemin Wen&amp;nbsp; 温合民
> > > &gt; wenhemin@apache.org
> > > &gt; --------------------
> > > &gt;
> > > &gt;
> > > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月22日周五
> 上午10:20写道：
> > > &gt;
> > > &gt; &amp;gt; Your are right, it can be solved the
> problem by modifying
> > the
> > > &gt; &amp;gt; number of server ssh
> connections.&amp;amp;nbsp;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; The real reason is that I want to control
> the&amp;amp;nbsp;
> > > number of SSH
> > > &gt; &amp;gt; connections and reuse the SSH connection,
> Because the
> > remote
> > > server is
> > > &gt; &amp;gt; managed by other team.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Otherwise, I think that we can also
> control the number of
> > > parallel
> > > &gt; tasks
> > > &gt; &amp;gt; in a workflow.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; It is a good idea that we can use the
> encapsulated ssh
> > > connection to
> > > &gt; get
> > > &gt; &amp;gt; clear
> > > &gt; &amp;gt; exception information&amp;amp;nbsp;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"Hemin Wen"<
> wenhemin@apache.org
> > > &amp;amp;gt;;
> > > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月21日(星期四)
> 中午11:07
> > > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<
> dev@dolphinscheduler.apache.org
> > > &amp;amp;gt;;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support
> SSH Task
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Thanks.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; About ssh connections pool,
> > > &gt; &amp;gt; 1. If the connection is not enough, Can it
> be solved by
> > > modifying the
> > > &gt; &amp;gt; number of server ssh connections.
> > > &gt; &amp;gt; 2. I think we have to look at whether a
> server's parallel
> > > ssh tasks
> > > &gt; will
> > > &gt; &amp;gt; reach the bottleneck of server connections.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; I think it is better to use the
> encapsulated ssh connection
> > > to get
> > > &gt; clear
> > > &gt; &amp;gt; exception information.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; --------------------
> > > &gt; &amp;gt; DolphinScheduler(Incubator) Commtter
> > > &gt; &amp;gt; Hemin Wen&amp;amp;nbsp; 温合民
> > > &gt; &amp;gt; wenhemin@apache.org
> > > &gt; &amp;gt; --------------------
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt;
> 于2020年5月20日周三 下午11:29写道：
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Thanks for your advise
> and suggestions. It may
> > > has some
> > > &gt; exceptions if
> > > &gt; &amp;gt; I
> > > &gt; &amp;gt; &amp;amp;gt; implement SSH Task as you
> said. Now, let me
> > > explain my ideas.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; e.g
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; My workflow DAG has 100
> or more task, it's all
> > > ssh task. I
> > > &gt; &amp;gt; have&amp;amp;amp;nbsp; two
> > > &gt; &amp;gt; &amp;amp;gt; questions.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 1、We must limit SSH
> connection and reuse the
> > > SSH connections,
> > > &gt; &amp;gt; otherwise
> > > &gt; &amp;gt; &amp;amp;gt; the remote server will be
> refused.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 2、If we config all of
> these task, we have a
> > lot
> > > of repeat
> > > &gt; work to do.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; By the way, the way I'm
> currently verifying is
> > > I use local
> > > &gt; params to
> > > &gt; &amp;gt; &amp;amp;gt; config the remote
> server's host,user,password
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> 发件人:&amp;amp;amp;nbsp;"lidong dai"<
> > > dailidong66@gmail.com
> > > &gt; &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > > &gt; &amp;gt; &amp;amp;gt;
> 收件人:&amp;amp;amp;nbsp;"dev"<
> > > dev@dolphinscheduler.apache.org
> > > &gt; &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> 主题:&amp;amp;amp;nbsp;Re: [Feature] Support SSH
> > > Task
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; hi, I see you will Jsch,
> does his license
> > > compitable with
> > > &gt; apache v2
> > > &gt; &amp;gt; &amp;amp;gt; license, I see the
> following words "
> > > &gt; &amp;gt; &amp;amp;gt; JSch 0.0.* was released
> under the GNU LGPL
> > > license. Later,
> > > &gt; we have
> > > &gt; &amp;gt; switched
> > > &gt; &amp;gt; &amp;amp;gt; over to a BSD-style
> license.
> > > &gt; &amp;gt; &amp;amp;gt; " on
> > > https://github.com/is/jsch/blob/master/LICENSE.txt .
> > > &gt; you know,
> > > &gt; &amp;gt; LGPL
> > > &gt; &amp;gt; &amp;amp;gt; license not compitable
> with apache license.
> > but
> > > BSD is ok ,
> > > &gt; so the
> > > &gt; &amp;gt; version
> > > &gt; &amp;gt; &amp;amp;gt; of Jsch is key point
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; second, Connection Pool
> is heavy resource,
> > > Please use
> > > &gt; carefully
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; and I'm a little
> confused, why there must be
> > > add a new
> > > &gt; &amp;gt; Table,&amp;amp;amp;nbsp; other
> > > &gt; &amp;gt; &amp;amp;gt; task
> > > &gt; &amp;gt; &amp;amp;gt; types also no seperate
> > table,&amp;amp;amp;nbsp;
> > > could you
> > > &gt; consider task_type
> > > &gt; &amp;gt; and
> > > &gt; &amp;gt; &amp;amp;gt; task_json
> > > &gt; &amp;gt; &amp;amp;gt; in t_ds_task_instance ?
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Best Regards
> > > &gt; &amp;gt; &amp;amp;gt; ---------------
> > > &gt; &amp;gt; &amp;amp;gt;
> DolphinScheduler(Incubator) PPMC
> > > &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬
> > > &gt; &amp;gt; &amp;amp;gt; dailidong66@gmail.com
> > > &gt; &amp;gt; &amp;amp;gt; ---------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 裴龙武 <peilongwu@qq.com
> &amp;amp;amp;gt;
> > > 于2020年5月20日周三 下午9:57写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I
> want to describe my idea.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> First, we must design a host
> > > table that's named
> > > &gt; &amp;gt; t_ds_ssh_host, e.g
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - id
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> name
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - ip
> / host
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> user
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> password
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> max_connection
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> create_time
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> update_time
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; -
> user_id
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Second, Shell can execute in
> > > worker server or
> > > &gt; remote host.
> > > &gt; &amp;gt; We can
> > > &gt; &amp;gt; &amp;amp;gt; choose a
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; host
> in host's list.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Third, when workflow running,
> > > we must maintain
> > > &gt; the remote
> > > &gt; &amp;gt; host's
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Finally, when workflow
> > finish,
> > > we must release
> > > &gt; the
> > > &gt; &amp;gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Why
> do we have to maintain a
> > > connection pool.
> > > &gt; Because it is
> > > &gt; &amp;gt; very easy
> > > &gt; &amp;gt; &amp;amp;gt; to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; get
> exceptions when it's has
> > > many SSH task.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 发件人:&amp;amp;amp;amp;nbsp;"lidong dai"<
> > > &gt; dailidong66@gmail.com
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 发送时间:&amp;amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 收件人:&amp;amp;amp;amp;nbsp;"dev"<
> > > &gt; dev@dolphinscheduler.apache.org
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 主题:&amp;amp;amp;amp;nbsp;Re:
> > > [Feature] Support SSH
> > > &gt; Task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; glad
> to hear that you will
> > > implement this
> > > &gt; feature
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Best
> Regards
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ---------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> DolphinScheduler(Incubator)
> > > PPMC
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Lidong Dai 代立冬
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> dailidong66@gmail.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ---------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 裴龙武 <
> peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;
> > > &gt; 于2020年5月20日周三 下午3:47写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; My code
> > > is not perfect yet. I
> > > &gt; will write a
> > > &gt; &amp;gt; detailed design
> > > &gt; &amp;gt; &amp;amp;gt; document.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Then
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; I will
> > > realize this feature
> > > &gt; about our
> > > &gt; &amp;gt; discussion result.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 发件人:&amp;amp;amp;amp;amp;nbsp;"wenhemin"<
> > > &gt; &amp;gt; whm_777@163.com
> &amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 发送时间:&amp;amp;amp;amp;amp;nbsp;2020年5月18日(星期一)
> 晚上7:50
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 收件人:&amp;amp;amp;amp;amp;nbsp;"裴龙武"<
> peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;;"dev"<
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > dev@dolphinscheduler.apache.org
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > 主题:&amp;amp;amp;amp;amp;nbsp;Re:
> > > &gt; [Feature] Support SSH
> > > &gt; &amp;gt; Task and Support
> > > &gt; &amp;gt; &amp;amp;gt; dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; Thanks
> > > for writing detailed
> > > &gt; documentation. I
> > > &gt; &amp;gt; think this is
> > > &gt; &amp;gt; &amp;amp;gt; also a
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> missing
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; feature
> > > of DS.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; About
> > the
> > > extension point:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 1.Can
> > ssh
> > > tasks be merged into
> > > &gt; shell tasks.
> > > &gt; &amp;gt; Essentially,
> > > &gt; &amp;gt; &amp;amp;gt; they all
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; shell
> > > commands.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 2.About
> > > dummy task, DS has the
> > > &gt; function of
> > > &gt; &amp;gt; disable nodes, I
> > > &gt; &amp;gt; &amp;amp;gt; do n’t
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; know
> if
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; this
> > > requirement is met.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; The
> > > script from AirFlow to
> > > &gt; Dolphin is great.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; 在
> > > &gt; 2020年5月18日，09:28，裴龙武 <
> > > &gt; &amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; OK, 3Q!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; First, I
> > > &gt; will ensure that open
> > > &gt; &amp;gt; source can use.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; Second, I
> > > &gt; think we must
> > > &gt; &amp;gt; discuss deeply. I write
> > > &gt; &amp;gt; &amp;amp;gt; a more
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> detailed
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > document.
> > > You can check the
> > > &gt; attachment. I also
> > > &gt; &amp;gt; send the
> > > &gt; &amp;gt; &amp;amp;gt; document to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > DaiLidong.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; Third,&amp;amp;amp;amp;amp;nbsp; I'll
> > > &gt; &amp;gt; give you the error of
> > > &gt; &amp;gt; &amp;amp;gt; not using SSH
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; ------------------ 原始邮件
> > > &gt; &amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; 发件人:
> > > &gt; "wenhemin"<
> > > &gt; &amp;gt; whm_777@163.com
> &amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; 发送时间:
> > > &gt; 2020年5月14日(星期四) 晚上7:26
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; 收件人:
> > > &gt; "裴龙武"<peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; 主题: Re:
> > > &gt; [Feature] Support SSH
> > > &gt; &amp;gt; Task and Support
> > > &gt; &amp;gt; &amp;amp;gt; dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; Great!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; I think,
> > > &gt; Can ssh tasks be
> > > &gt; &amp;gt; merged into shell
> > > &gt; &amp;gt; &amp;amp;gt;
> tasks,&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; script
> > > locally or remotely,
> > > &gt; Configure on the
> > > &gt; &amp;gt; front end.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; About ssh
> > > &gt; connect pool, I did
> > > &gt; &amp;gt; not find it
> > > &gt; &amp;gt; &amp;amp;gt; necessary to use
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt; BTW, Look
> > > &gt; at the code to
> > > &gt; &amp;gt; introduce additional
> > > &gt; &amp;gt; &amp;amp;gt; jar packages,
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; You
> also
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; need to
> > > ensure that open
> > > &gt; source can use the
> > > &gt; &amp;gt; license of this
> > > &gt; &amp;gt; &amp;amp;gt; jar
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> package.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 在
> > > &gt; &amp;gt; 2020年5月14日，16:20，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 1. The
> > > &gt; &amp;gt; priority between these tasks
> > > &gt; &amp;gt; &amp;amp;gt; is also depended
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; on
> the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; dolphin
> > > DAG define. When the
> > > &gt; front task is not
> > > &gt; &amp;gt; finished, it
> > > &gt; &amp;gt; &amp;amp;gt; not
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; next
> > task.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 2. I extend
> > > &gt; &amp;gt; ssh task. I also use
> > > &gt; &amp;gt; &amp;amp;gt; local params to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> config ssh host,
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; user and
> > > password.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; E.g:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; public static
> > > &gt; &amp;gt; AbstractTask
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> newTask(TaskExecutionContext
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > taskExecutionContext, Logger
> > > &gt; logger)
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; throws
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> IllegalArgumentException {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; Boolean
> > > &gt; &amp;gt; &amp;amp;gt; enable =
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; if
> > > &gt; &amp;gt; &amp;amp;gt; (enable != null
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;
> > > &gt; enable == false ) {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; DummyTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; switch
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case SHELL:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; ShellTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> PROCEDURE:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; ProcedureTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case SQL:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > SqlTask(taskExecutionContext,
> > > &gt; logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case MR:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; MapReduceTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case SPARK:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; SparkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case FLINK:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; FlinkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case PYTHON:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; PythonTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case HTTP:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > HttpTask(taskExecutionContext,
> > > &gt; logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case DATAX:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; DataxTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case SQOOP:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; SqoopTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; case SSH:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > SSHTask(taskExecutionContext,
> > > &gt; logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; default:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> logger.error("unsupport task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; type:
> > {}",
> > > &gt; taskExecutionContext.getTaskType());
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> throw new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > IllegalArgumentException("not
> > > &gt; support task
> > > &gt; &amp;gt; type");
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 3. I am not
> > > &gt; &amp;gt; sure that it supports
> > > &gt; &amp;gt; &amp;amp;gt; window or not.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 发件人:
> > > &gt; &amp;gt; "wenhemin"<whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> whm_777@163.com
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 发送时间:
> > > &gt; &amp;gt; 2020年5月14日(星期四) 下午3:46
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 收件人: "裴龙武"<
> > > &gt; &amp;gt; peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; <mailto:peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 主题: Re:
> > > &gt; &amp;gt; [Feature] Support SSH Task
> > > &gt; &amp;gt; &amp;amp;gt; and Support dummy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task
> like
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; Sorry, My
> > > &gt; &amp;gt; previous description is
> > > &gt; &amp;gt; &amp;amp;gt; not very clear.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; I want to ask
> > > &gt; &amp;gt; some questions:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 1.How to
> > > &gt; &amp;gt; control the priority
> > > &gt; &amp;gt; &amp;amp;gt; between ssh tasks?
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> There may be
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; some ssh
> > > tasks that have been
> > > &gt; waiting for
> > > &gt; &amp;gt; execution.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 2.I understand
> > > &gt; &amp;gt; what you want to
> > > &gt; &amp;gt; &amp;amp;gt; solve is the problem
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; of
> executing
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; remote
> > > ssh scripts in batches.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; So, not
> > > &gt; &amp;gt; &amp;amp;gt; sure how to use this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> function.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 3.I don't know
> > > &gt; &amp;gt; if this supports
> > > &gt; &amp;gt; &amp;amp;gt; windows system.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > 在
> > > &gt; 2020年5月13日，20:56，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > I
> > > use
> > > &gt; spin lock. Here
> > > &gt; &amp;gt; &amp;amp;gt; is my code. Of course
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ,
> it's not
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; perfect.
> > > I just do a test. To
> > > &gt; my surprise, it
> > > &gt; &amp;gt; is the result
> > > &gt; &amp;gt; &amp;amp;gt; of the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > execution
> > > is the same as the
> > > &gt; AirFlow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和
> AirFlow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 相同的结果。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > 发件人:
> > > &gt; "whm_777"<
> > > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > 发送时间:
> > > &gt; 2020年5月13日(星期三)
> > > &gt; &amp;gt; &amp;amp;gt; 晚上7:21
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > 收件人:
> > > &gt; "裴龙武"<
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > 主题: Re:
> > > &gt; [Feature]
> > > &gt; &amp;gt; &amp;amp;gt; Support SSH Task and
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; like
> > > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > You can
> > > &gt; modify the
> > > &gt; &amp;gt; &amp;amp;gt; maximum number of linux
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh
> connections.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > If
> > > use
> > > &gt; ssh connection
> > > &gt; &amp;gt; &amp;amp;gt; pool, How to control
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the
> priority of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; ssh?
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; 在
> > > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，18:01，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; First 3Q，
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> I&amp;amp;amp;amp;amp;nbsp; use more than 100
> > > task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> node. But SSH connections
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; are
> > > limited.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> <330BE502@F7F80E73.76C5BB5E.jpg
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; 发件人:
> > > &gt; &amp;gt; &amp;amp;gt; "whm_777"<whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> <mailto:whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; 发送时间:
> > > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日(星期三) 下午5:50
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; 收件人: "裴龙武"<
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; 主题: Re:
> > > &gt; &amp;gt; &amp;amp;gt; [Feature] Support SSH Task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; and
> Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; like
> > > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; E.g.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; rtn_code=`ssh -o
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ServerAliveInterval=60 -p
> > xxxx
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > root@xxx.xxx.xxx.xxx <mailto:
> > > &gt; &amp;gt; root@xxx.xxx.xxx.xxx
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; ‘shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > command&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;/dev/null
> > > &gt; &amp;gt; &amp;amp;gt;
> > > 2&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;1;
> echo $?'`
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; if [
> > > &gt; &amp;gt; &amp;amp;gt; "$rtn_code" -eq 0 ]; then
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo
> "成功"
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; exit
> 0
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; else
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo
> "失败"
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; exit
> 1
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; fi
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; Batch shell
> > > &gt; &amp;gt; &amp;amp;gt; command is not supported.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; Multiple
> > > &gt; &amp;gt; &amp;amp;gt; servers can be split into
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> multiple task nodes.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Could you give me a
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> example，3Q. 能否给我一个例子，谢谢！
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; By the way, I have more
> than
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 100
> tasks in one DAG.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; These
> > > tasks connect two other
> > > &gt; server to
> > > &gt; &amp;gt; execute. So SSH
> > > &gt; &amp;gt; &amp;amp;gt; tasks must
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; have
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; pool to
> > > manager. Now I use
> > > &gt; JSch and realize a
> > > &gt; &amp;gt; simple pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 任务，这些任务连接两台任务服务器进行任务执行。所以
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; SSH
> > > 任务进行连接时，必须使用连接池进行管理。当前我使用
> > > &gt; JSch，并实现了一个简单的连接池。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 发件人: "wenhemin"<
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> whm_777@163.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > whm_777@163.com
> > > &gt; &amp;gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 发送时间: 2020年5月13日(星期三)
> 下午5:24
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 收件人: "dev"<
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > dev@dolphinscheduler.apache.org <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; dev@dolphinscheduler.apache.org
> > > &gt; &amp;gt;
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 主题: Re: [Feature] Support
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH
> Task and Support dummy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; task
> > like
> > > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; The shell node is supports
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> remote calling, and get
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; the
> > > remote command result code.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; 在
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 2020年5月13日，15:16，裴龙武 <
> > > peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; <mailto:
> > > peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Dear ALL：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Support Linux SSH
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Task
> 支持 Linux SSH 任务
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 场景描述：当前项目中，工作流的任务的目标是执行不同服务器
> > > Shell 脚本，Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > 脚本是保存在业务服务器的固定目录。当 Worker
> > > &gt; &amp;gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> project, the workflow's tasks
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; want to
> > > execute shell scripts
> > > &gt; where are in
> > > &gt; &amp;gt; different server's
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> different
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > directory. When worker execute
> > > &gt; these shell
> > > &gt; &amp;gt; scripts, it must
> > > &gt; &amp;gt; &amp;amp;gt; use the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; same
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; user to
> > > login these server.
> > > &gt; Also, the worker
> > > &gt; &amp;gt; can get the
> > > &gt; &amp;gt; &amp;amp;gt; executing
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> state of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; these
> > > server. We can config
> > > &gt; these server 's
> > > &gt; &amp;gt; host,user and
> > > &gt; &amp;gt; &amp;amp;gt; password.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; SSH Task is very
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> useful for most user SSH
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > 任务对大多数用户是非常有用的
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > > &gt; &amp;gt; Worker
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; In
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> dolphinscheduler, the most
> > > executing tasks
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; are in
> > > different servers who
> > > &gt; are not workers.
> > > &gt; &amp;gt; These servers
> > > &gt; &amp;gt; &amp;amp;gt; also have
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; their
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > different
> > > fixed services. We
> > > &gt; just have to pass
> > > &gt; &amp;gt; different
> > > &gt; &amp;gt; &amp;amp;gt; parameters to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; schedule
> > > these shell scripts
> > > &gt; to execute.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to
> execute ssh script Python
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > 有固定的工具包，可执行这些SSH Shell 脚本
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Python
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 有一个可执行远程服务器SSH
> > > Shell脚本的模块，其名字为：paramiko。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; that
> can execute SSH Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; script.
> > > It's paramiko.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Others 其他内容
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; I found this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> described in previous
> > feature,
> > > but
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; it was
> > > relatively simple.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Feature URL
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; 另外，我通过 Shell Task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; In addition, it is
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; very
> inconvenient for me to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; perform
> > > remote tasks through
> > > &gt; Shell Task. Here
> > > &gt; &amp;gt; is my script.
> > > &gt; &amp;gt; &amp;amp;gt; I don't
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; know
> if
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; there's
> > a
> > > better way.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; sshpass -p
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 'password' ssh user@host
> > echo
> > > 'ssh
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; success'
> > > echo 'Hello World'
> > > &gt; &amp;gt; -&amp;amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > /home/dolphinscheduler/test/hello.txt
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; echo
> > 'end'
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> airflow 支持像 Airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 中的虚拟任务
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG
> 文件，DAG
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Dummy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> project, it has a productized
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; DAG
> > file.
> > > The file contains
> > > &gt; different modules,
> > > &gt; &amp;gt; some of which
> > > &gt; &amp;gt; &amp;amp;gt; are
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > interdependent and some of
> > > &gt; which are not. When
> > > &gt; &amp;gt; customers
> > > &gt; &amp;gt; &amp;amp;gt; purchase
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> different
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; modules,
> > > we need to set some
> > > &gt; tasks as dummy
> > > &gt; &amp;gt; tasks, which
> > > &gt; &amp;gt; &amp;amp;gt; some modules
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; not
> > > purchased and the
> > > &gt; purchased module is not
> > > &gt; &amp;gt; dependent.
> > > &gt; &amp;gt; &amp;amp;gt; Because of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; setting,
> > > these dummy tasks are
> > > &gt; actually not
> > > &gt; &amp;gt; executed. The
> > > &gt; &amp;gt; &amp;amp;gt; benefits of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; setup
> > are
> > > product unity and
> > > &gt; diagram integrity.
> > > &gt; &amp;gt; In airflow,
> > > &gt; &amp;gt; &amp;amp;gt; these task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; execute
> > > by dummy operator.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; ** Realize 实现方式**
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Dummy Task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; dummy
> > > 时，实际的任务不执行，执行 Dummy Task。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; Dummy Task is easy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to
> realize, but it need to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; use with
> > > other different
> > > &gt; tasks. When the task's
> > > &gt; &amp;gt; executed
> > > &gt; &amp;gt; &amp;amp;gt; type is set
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; dummy
> > > type, the task are
> > > &gt; executed as a dummy
> > > &gt; &amp;gt; task and the
> > > &gt; &amp;gt; &amp;amp;gt; real task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; is
> not
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > executed.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;gt; By the way，I
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> already realize these
> > > &gt; two&amp;amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; features
> > > in my fork
> > > &gt; &amp;gt;
> branch.&amp;amp;amp;amp;amp;amp;nbsp;Whether the
> > > &gt; &amp;gt; &amp;amp;gt; follow-up
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> release can be
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > supported
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt;
> > >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> <SSHClient.java&amp;amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > > &amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt;
> <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;amp;gt;
> >
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
OK, I'll try to do it by the way we talked about it.
------------------ 原始邮件 ------------------
发件人: "lidong dai"<dailidong66@gmail.com>;
发送时间: 2020年5月27日(星期三) 晚上7:51
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
I think your idea is good, haha
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
Hemin Wen <wenhemin@apache.org> 于2020年5月27日周三 上午11:53写道：
> Is it possible to extend the data source and increase the ssh type.
> I look at the data structure of DB is ok.
>
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月27日周三 上午10:55写道：
>
> > Good idea! It's can solve the exceptions where the numbers of SSH
> > connections exceeds.
> >
> >
> > We also have another problem of how to config remote server easily when
> we
> > have hundreds of tasks.
> >
> >
> > Now I use the local param to config remote server's IP,username,password.
> > But every task needs to config.
> >
> >
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> > 发送时间:&nbsp;2020年5月26日(星期二) 晚上6:38
> > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task
> >
> >
> >
> > good idea, I think "Control the number of task's concurrent" may be have
> > some other methods,&nbsp; for now, we don't have this feature, but when I
> > communicate with the team of China Unicom， they provide a new solve
> > solution：
> > &nbsp; we can add a task concurrent page, you can set a variable value
> like
> > 'ETL_JOB' =10 and so on, then give the task type add a option for
> choosing
> > ETL_JOB or other variable, and there may be many task also choose ETL_JOB
> > variable.&nbsp; when these tasks begin to running at the same time, the
> > Master
> > Server will check whether a task begin to run or wait(if the running
> tasks
> > num = 10)
> >
> >
> > Best egards
> > ---------------
> > DolphinScheduler(Incubator) PPMC
> > Lidong Dai 代立冬
> > dailidong66@gmail.com
> > ---------------
> >
> >
> > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月26日周二 下午2:34写道：
> >
> > &gt; There is only two ways that I can think to do about this.
> > &gt;
> > &gt;
> > &gt; 1. Simple SSH connect pool
> > &gt; 2. Control the number of task's concurrent
> > &gt;
> > &gt;
> > &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:&amp;nbsp;"Hemin Wen"<wenhemin@apache.org&amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2020年5月22日(星期五) 晚上7:29
> > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org&amp;gt;;
> > &gt;
> > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > &gt;
> > &gt;
> > &gt;
> > &gt; What other friends think about this discuss
> > &gt;
> > &gt;
> > &gt; --------------------
> > &gt; DolphinScheduler(Incubator) Commtter
> > &gt; Hemin Wen&amp;nbsp; 温合民
> > &gt; wenhemin@apache.org
> > &gt; --------------------
> > &gt;
> > &gt;
> > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月22日周五 上午10:20写道：
> > &gt;
> > &gt; &amp;gt; Your are right, it can be solved the problem by modifying
> the
> > &gt; &amp;gt; number of server ssh connections.&amp;amp;nbsp;
> > &gt; &amp;gt;
> > &gt; &amp;gt; The real reason is that I want to control the&amp;amp;nbsp;
> > number of SSH
> > &gt; &amp;gt; connections and reuse the SSH connection, Because the
> remote
> > server is
> > &gt; &amp;gt; managed by other team.
> > &gt; &amp;gt;
> > &gt; &amp;gt; Otherwise, I think that we can also control the number of
> > parallel
> > &gt; tasks
> > &gt; &amp;gt; in a workflow.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; It is a good idea that we can use the encapsulated ssh
> > connection to
> > &gt; get
> > &gt; &amp;gt; clear
> > &gt; &amp;gt; exception information&amp;amp;nbsp;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"Hemin Wen"<wenhemin@apache.org
> > &amp;amp;gt;;
> > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月21日(星期四) 中午11:07
> > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> > &amp;amp;gt;;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH Task
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Thanks.
> > &gt; &amp;gt;
> > &gt; &amp;gt; About ssh connections pool,
> > &gt; &amp;gt; 1. If the connection is not enough, Can it be solved by
> > modifying the
> > &gt; &amp;gt; number of server ssh connections.
> > &gt; &amp;gt; 2. I think we have to look at whether a server's parallel
> > ssh tasks
> > &gt; will
> > &gt; &amp;gt; reach the bottleneck of server connections.
> > &gt; &amp;gt;
> > &gt; &amp;gt; I think it is better to use the encapsulated ssh connection
> > to get
> > &gt; clear
> > &gt; &amp;gt; exception information.
> > &gt; &amp;gt;
> > &gt; &amp;gt; --------------------
> > &gt; &amp;gt; DolphinScheduler(Incubator) Commtter
> > &gt; &amp;gt; Hemin Wen&amp;amp;nbsp; 温合民
> > &gt; &amp;gt; wenhemin@apache.org
> > &gt; &amp;gt; --------------------
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt; 于2020年5月20日周三 下午11:29写道：
> > &gt; &amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Thanks for your advise and suggestions. It may
> > has some
> > &gt; exceptions if
> > &gt; &amp;gt; I
> > &gt; &amp;gt; &amp;amp;gt; implement SSH Task as you said. Now, let me
> > explain my ideas.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; e.g
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; My workflow DAG has 100 or more task, it's all
> > ssh task. I
> > &gt; &amp;gt; have&amp;amp;amp;nbsp; two
> > &gt; &amp;gt; &amp;amp;gt; questions.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 1、We must limit SSH connection and reuse the
> > SSH connections,
> > &gt; &amp;gt; otherwise
> > &gt; &amp;gt; &amp;amp;gt; the remote server will be refused.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 2、If we config all of these task, we have a
> lot
> > of repeat
> > &gt; work to do.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; By the way, the way I'm currently verifying is
> > I use local
> > &gt; params to
> > &gt; &amp;gt; &amp;amp;gt; config the remote server's host,user,password
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt; 发件人:&amp;amp;amp;nbsp;"lidong dai"<
> > dailidong66@gmail.com
> > &gt; &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > &gt; &amp;gt; &amp;amp;gt; 收件人:&amp;amp;amp;nbsp;"dev"<
> > dev@dolphinscheduler.apache.org
> > &gt; &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re: [Feature] Support SSH
> > Task
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; hi, I see you will Jsch, does his license
> > compitable with
> > &gt; apache v2
> > &gt; &amp;gt; &amp;amp;gt; license, I see the following words "
> > &gt; &amp;gt; &amp;amp;gt; JSch 0.0.* was released under the GNU LGPL
> > license. Later,
> > &gt; we have
> > &gt; &amp;gt; switched
> > &gt; &amp;gt; &amp;amp;gt; over to a BSD-style license.
> > &gt; &amp;gt; &amp;amp;gt; " on
> > https://github.com/is/jsch/blob/master/LICENSE.txt .
> > &gt; you know,
> > &gt; &amp;gt; LGPL
> > &gt; &amp;gt; &amp;amp;gt; license not compitable with apache license.
> but
> > BSD is ok ,
> > &gt; so the
> > &gt; &amp;gt; version
> > &gt; &amp;gt; &amp;amp;gt; of Jsch is key point
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; second, Connection Pool is heavy resource,
> > Please use
> > &gt; carefully
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; and I'm a little confused, why there must be
> > add a new
> > &gt; &amp;gt; Table,&amp;amp;amp;nbsp; other
> > &gt; &amp;gt; &amp;amp;gt; task
> > &gt; &amp;gt; &amp;amp;gt; types also no seperate
> table,&amp;amp;amp;nbsp;
> > could you
> > &gt; consider task_type
> > &gt; &amp;gt; and
> > &gt; &amp;gt; &amp;amp;gt; task_json
> > &gt; &amp;gt; &amp;amp;gt; in t_ds_task_instance ?
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Best Regards
> > &gt; &amp;gt; &amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt; DolphinScheduler(Incubator) PPMC
> > &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬
> > &gt; &amp;gt; &amp;amp;gt; dailidong66@gmail.com
> > &gt; &amp;gt; &amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;amp;gt;
> > 于2020年5月20日周三 下午9:57写道：
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I want to describe my idea.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, we must design a host
> > table that's named
> > &gt; &amp;gt; t_ds_ssh_host, e.g
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - id
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - name
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - ip / host
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - user
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - password
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - max_connection
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - create_time
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - update_time
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; - user_id
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second, Shell can execute in
> > worker server or
> > &gt; remote host.
> > &gt; &amp;gt; We can
> > &gt; &amp;gt; &amp;amp;gt; choose a
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; host in host's list.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Third, when workflow running,
> > we must maintain
> > &gt; the remote
> > &gt; &amp;gt; host's
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Finally, when workflow
> finish,
> > we must release
> > &gt; the
> > &gt; &amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Why do we have to maintain a
> > connection pool.
> > &gt; Because it is
> > &gt; &amp;gt; very easy
> > &gt; &amp;gt; &amp;amp;gt; to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; get exceptions when it's has
> > many SSH task.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 发件人:&amp;amp;amp;amp;nbsp;"lidong dai"<
> > &gt; dailidong66@gmail.com
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 发送时间:&amp;amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 收件人:&amp;amp;amp;amp;nbsp;"dev"<
> > &gt; dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题:&amp;amp;amp;amp;nbsp;Re:
> > [Feature] Support SSH
> > &gt; Task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; glad to hear that you will
> > implement this
> > &gt; feature
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Best Regards
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DolphinScheduler(Incubator)
> > PPMC
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Lidong Dai 代立冬
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dailidong66@gmail.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 裴龙武 <peilongwu@qq.com
> > &amp;amp;amp;amp;gt;
> > &gt; 于2020年5月20日周三 下午3:47写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; My code
> > is not perfect yet. I
> > &gt; will write a
> > &gt; &amp;gt; detailed design
> > &gt; &amp;gt; &amp;amp;gt; document.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Then
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; I will
> > realize this feature
> > &gt; about our
> > &gt; &amp;gt; discussion result.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 发件人:&amp;amp;amp;amp;amp;nbsp;"wenhemin"<
> > &gt; &amp;gt; whm_777@163.com&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 发送时间:&amp;amp;amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 收件人:&amp;amp;amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;;"dev"<
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > 主题:&amp;amp;amp;amp;amp;nbsp;Re:
> > &gt; [Feature] Support SSH
> > &gt; &amp;gt; Task and Support
> > &gt; &amp;gt; &amp;amp;gt; dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; Thanks
> > for writing detailed
> > &gt; documentation. I
> > &gt; &amp;gt; think this is
> > &gt; &amp;gt; &amp;amp;gt; also a
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; missing
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; feature
> > of DS.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; About
> the
> > extension point:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 1.Can
> ssh
> > tasks be merged into
> > &gt; shell tasks.
> > &gt; &amp;gt; Essentially,
> > &gt; &amp;gt; &amp;amp;gt; they all
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; shell
> > commands.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 2.About
> > dummy task, DS has the
> > &gt; function of
> > &gt; &amp;gt; disable nodes, I
> > &gt; &amp;gt; &amp;amp;gt; do n’t
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; this
> > requirement is met.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; The
> > script from AirFlow to
> > &gt; Dolphin is great.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; 在
> > &gt; 2020年5月18日，09:28，裴龙武 <
> > &gt; &amp;gt; peilongwu@qq.com&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; OK, 3Q!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; First, I
> > &gt; will ensure that open
> > &gt; &amp;gt; source can use.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; Second, I
> > &gt; think we must
> > &gt; &amp;gt; discuss deeply. I write
> > &gt; &amp;gt; &amp;amp;gt; a more
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; detailed
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> document.
> > You can check the
> > &gt; attachment. I also
> > &gt; &amp;gt; send the
> > &gt; &amp;gt; &amp;amp;gt; document to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> DaiLidong.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; Third,&amp;amp;amp;amp;amp;nbsp; I'll
> > &gt; &amp;gt; give you the error of
> > &gt; &amp;gt; &amp;amp;gt; not using SSH
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; ------------------ 原始邮件
> > &gt; &amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; 发件人:
> > &gt; "wenhemin"<
> > &gt; &amp;gt; whm_777@163.com&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; 发送时间:
> > &gt; 2020年5月14日(星期四) 晚上7:26
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; 收件人:
> > &gt; "裴龙武"<peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; 主题: Re:
> > &gt; [Feature] Support SSH
> > &gt; &amp;gt; Task and Support
> > &gt; &amp;gt; &amp;amp;gt; dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; Great!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; I think,
> > &gt; Can ssh tasks be
> > &gt; &amp;gt; merged into shell
> > &gt; &amp;gt; &amp;amp;gt; tasks,&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; script
> > locally or remotely,
> > &gt; Configure on the
> > &gt; &amp;gt; front end.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; About ssh
> > &gt; connect pool, I did
> > &gt; &amp;gt; not find it
> > &gt; &amp;gt; &amp;amp;gt; necessary to use
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt; BTW, Look
> > &gt; at the code to
> > &gt; &amp;gt; introduce additional
> > &gt; &amp;gt; &amp;amp;gt; jar packages,
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; You also
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; need to
> > ensure that open
> > &gt; source can use the
> > &gt; &amp;gt; license of this
> > &gt; &amp;gt; &amp;amp;gt; jar
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; package.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 在
> > &gt; &amp;gt; 2020年5月14日，16:20，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 1. The
> > &gt; &amp;gt; priority between these tasks
> > &gt; &amp;gt; &amp;amp;gt; is also depended
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; on the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; dolphin
> > DAG define. When the
> > &gt; front task is not
> > &gt; &amp;gt; finished, it
> > &gt; &amp;gt; &amp;amp;gt; not
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; next
> task.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 2. I extend
> > &gt; &amp;gt; ssh task. I also use
> > &gt; &amp;gt; &amp;amp;gt; local params to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; config ssh host,
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; user and
> > password.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; E.g:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; public static
> > &gt; &amp;gt; AbstractTask
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; newTask(TaskExecutionContext
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > taskExecutionContext, Logger
> > &gt; logger)
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; throws
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; IllegalArgumentException {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; Boolean
> > &gt; &amp;gt; &amp;amp;gt; enable =
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; if
> > &gt; &amp;gt; &amp;amp;gt; (enable != null
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;
> > &gt; enable == false ) {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; DummyTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; switch
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case SHELL:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; ShellTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; PROCEDURE:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; ProcedureTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case SQL:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > SqlTask(taskExecutionContext,
> > &gt; logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case MR:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; MapReduceTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case SPARK:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; SparkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case FLINK:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; FlinkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case PYTHON:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; PythonTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case HTTP:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > HttpTask(taskExecutionContext,
> > &gt; logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case DATAX:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; DataxTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case SQOOP:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; SqoopTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; case SSH:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > SSHTask(taskExecutionContext,
> > &gt; logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; default:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; logger.error("unsupport task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; type:
> {}",
> > &gt; taskExecutionContext.getTaskType());
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; throw new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > IllegalArgumentException("not
> > &gt; support task
> > &gt; &amp;gt; type");
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 3. I am not
> > &gt; &amp;gt; sure that it supports
> > &gt; &amp;gt; &amp;amp;gt; window or not.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 发件人:
> > &gt; &amp;gt; "wenhemin"<whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; whm_777@163.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 发送时间:
> > &gt; &amp;gt; 2020年5月14日(星期四) 下午3:46
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 收件人: "裴龙武"<
> > &gt; &amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; <mailto:peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 主题: Re:
> > &gt; &amp;gt; [Feature] Support SSH Task
> > &gt; &amp;gt; &amp;amp;gt; and Support dummy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task like
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; Sorry, My
> > &gt; &amp;gt; previous description is
> > &gt; &amp;gt; &amp;amp;gt; not very clear.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; I want to ask
> > &gt; &amp;gt; some questions:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 1.How to
> > &gt; &amp;gt; control the priority
> > &gt; &amp;gt; &amp;amp;gt; between ssh tasks?
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; There may be
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; some ssh
> > tasks that have been
> > &gt; waiting for
> > &gt; &amp;gt; execution.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 2.I understand
> > &gt; &amp;gt; what you want to
> > &gt; &amp;gt; &amp;amp;gt; solve is the problem
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; of executing
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; remote
> > ssh scripts in batches.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; So, not
> > &gt; &amp;gt; &amp;amp;gt; sure how to use this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; function.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt; 3.I don't know
> > &gt; &amp;gt; if this supports
> > &gt; &amp;gt; &amp;amp;gt; windows system.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> 在
> > &gt; 2020年5月13日，20:56，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> I
> > use
> > &gt; spin lock. Here
> > &gt; &amp;gt; &amp;amp;gt; is my code. Of course
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; , it's not
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; perfect.
> > I just do a test. To
> > &gt; my surprise, it
> > &gt; &amp;gt; is the result
> > &gt; &amp;gt; &amp;amp;gt; of the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> execution
> > is the same as the
> > &gt; AirFlow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 相同的结果。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > 发件人:
> > &gt; "whm_777"<
> > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > 发送时间:
> > &gt; 2020年5月13日(星期三)
> > &gt; &amp;gt; &amp;amp;gt; 晚上7:21
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > 收件人:
> > &gt; "裴龙武"<
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > 主题: Re:
> > &gt; [Feature]
> > &gt; &amp;gt; &amp;amp;gt; Support SSH Task and
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; like
> > airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > You can
> > &gt; modify the
> > &gt; &amp;gt; &amp;amp;gt; maximum number of linux
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh connections.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> If
> > use
> > &gt; ssh connection
> > &gt; &amp;gt; &amp;amp;gt; pool, How to control
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the priority of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; ssh?
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; 在
> > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，18:01，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; First 3Q，
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; I&amp;amp;amp;amp;amp;nbsp; use more than 100
> > task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; node. But SSH connections
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; are
> > limited.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; <330BE502@F7F80E73.76C5BB5E.jpg
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; 发件人:
> > &gt; &amp;gt; &amp;amp;gt; "whm_777"<whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto:whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; 发送时间:
> > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日(星期三) 下午5:50
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; 收件人: "裴龙武"<
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; 主题: Re:
> > &gt; &amp;gt; &amp;amp;gt; [Feature] Support SSH Task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; and Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; like
> > airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; E.g.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; rtn_code=`ssh -o
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ServerAliveInterval=60 -p
> xxxx
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > root@xxx.xxx.xxx.xxx <mailto:
> > &gt; &amp;gt; root@xxx.xxx.xxx.xxx&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; ‘shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > command&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;/dev/null
> > &gt; &amp;gt; &amp;amp;gt;
> > 2&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;1; echo $?'`
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; if [
> > &gt; &amp;gt; &amp;amp;gt; "$rtn_code" -eq 0 ]; then
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo "成功"
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; exit 0
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; else
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo "失败"
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; exit 1
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; fi
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; Batch shell
> > &gt; &amp;gt; &amp;amp;gt; command is not supported.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; Multiple
> > &gt; &amp;gt; &amp;amp;gt; servers can be split into
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; multiple task nodes.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Could you give me a
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; example，3Q. 能否给我一个例子，谢谢！
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; By the way, I have more than
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 100 tasks in one DAG.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; These
> > tasks connect two other
> > &gt; server to
> > &gt; &amp;gt; execute. So SSH
> > &gt; &amp;gt; &amp;amp;gt; tasks must
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; have
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; pool to
> > manager. Now I use
> > &gt; JSch and realize a
> > &gt; &amp;gt; simple pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; SSH
> > 任务进行连接时，必须使用连接池进行管理。当前我使用
> > &gt; JSch，并实现了一个简单的连接池。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 发件人: "wenhemin"<
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > whm_777@163.com
> > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 收件人: "dev"<
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > dev@dolphinscheduler.apache.org <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 主题: Re: [Feature] Support
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH Task and Support dummy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; task
> like
> > airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; The shell node is supports
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; remote calling, and get
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; the
> > remote command result code.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; 在
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 2020年5月13日，15:16，裴龙武 <
> > peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; <mailto:
> > peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Dear ALL：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Support Linux SSH
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Task 支持 Linux SSH 任务
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器
> > Shell 脚本，Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > 脚本是保存在业务服务器的固定目录。当 Worker
> > &gt; &amp;gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; project, the workflow's tasks
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; want to
> > execute shell scripts
> > &gt; where are in
> > &gt; &amp;gt; different server's
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > directory. When worker execute
> > &gt; these shell
> > &gt; &amp;gt; scripts, it must
> > &gt; &amp;gt; &amp;amp;gt; use the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; same
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; user to
> > login these server.
> > &gt; Also, the worker
> > &gt; &amp;gt; can get the
> > &gt; &amp;gt; &amp;amp;gt; executing
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; state of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; these
> > server. We can config
> > &gt; these server 's
> > &gt; &amp;gt; host,user and
> > &gt; &amp;gt; &amp;amp;gt; password.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; SSH Task is very
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; useful for most user SSH
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > 任务对大多数用户是非常有用的
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > &gt; &amp;gt; Worker
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; In
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dolphinscheduler, the most
> > executing tasks
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; are in
> > different servers who
> > &gt; are not workers.
> > &gt; &amp;gt; These servers
> > &gt; &amp;gt; &amp;amp;gt; also have
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; their
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> different
> > fixed services. We
> > &gt; just have to pass
> > &gt; &amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; parameters to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; schedule
> > these shell scripts
> > &gt; to execute.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to execute ssh script Python
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Python
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 有一个可执行远程服务器SSH
> > Shell脚本的模块，其名字为：paramiko。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; that can execute SSH Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; script.
> > It's paramiko.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Others 其他内容
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; I found this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; described in previous
> feature,
> > but
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; it was
> > relatively simple.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Feature URL
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; 另外，我通过 Shell Task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; In addition, it is
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; very inconvenient for me to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; perform
> > remote tasks through
> > &gt; Shell Task. Here
> > &gt; &amp;gt; is my script.
> > &gt; &amp;gt; &amp;amp;gt; I don't
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; there's
> a
> > better way.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; sshpass -p
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 'password' ssh user@host
> echo
> > 'ssh
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; success'
> > echo 'Hello World'
> > &gt; &amp;gt; -&amp;amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > /home/dolphinscheduler/test/hello.txt
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; echo
> 'end'
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like airflow 支持像 Airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 中的虚拟任务
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG 文件，DAG
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Dummy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> > Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; project, it has a productized
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; DAG
> file.
> > The file contains
> > &gt; different modules,
> > &gt; &amp;gt; some of which
> > &gt; &amp;gt; &amp;amp;gt; are
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > interdependent and some of
> > &gt; which are not. When
> > &gt; &amp;gt; customers
> > &gt; &amp;gt; &amp;amp;gt; purchase
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; modules,
> > we need to set some
> > &gt; tasks as dummy
> > &gt; &amp;gt; tasks, which
> > &gt; &amp;gt; &amp;amp;gt; some modules
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; not
> > purchased and the
> > &gt; purchased module is not
> > &gt; &amp;gt; dependent.
> > &gt; &amp;gt; &amp;amp;gt; Because of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; setting,
> > these dummy tasks are
> > &gt; actually not
> > &gt; &amp;gt; executed. The
> > &gt; &amp;gt; &amp;amp;gt; benefits of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; setup
> are
> > product unity and
> > &gt; diagram integrity.
> > &gt; &amp;gt; In airflow,
> > &gt; &amp;gt; &amp;amp;gt; these task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; execute
> > by dummy operator.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; ** Realize 实现方式**
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Dummy Task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; dummy
> > 时，实际的任务不执行，执行 Dummy Task。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; Dummy Task is easy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to realize, but it need to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; use with
> > other different
> > &gt; tasks. When the task's
> > &gt; &amp;gt; executed
> > &gt; &amp;gt; &amp;amp;gt; type is set
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; dummy
> > type, the task are
> > &gt; executed as a dummy
> > &gt; &amp;gt; task and the
> > &gt; &amp;gt; &amp;amp;gt; real task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; is not
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> executed.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;gt; By the way，I
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; already realize these
> > &gt; two&amp;amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; features
> > in my fork
> > &gt; &amp;gt; branch.&amp;amp;amp;amp;amp;amp;nbsp;Whether the
> > &gt; &amp;gt; &amp;amp;gt; follow-up
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; release can be
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> supported
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt;
> > &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> <SSHClient.java&amp;amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> > &amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;amp;gt;
>
Re: [Feature] Support SSH Task
Posted by lidong dai <da...@gmail.com>.
I think your idea is good, haha
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
Hemin Wen <we...@apache.org> 于2020年5月27日周三 上午11:53写道：
> Is it possible to extend the data source and increase the ssh type.
> I look at the data structure of DB is ok.
>
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <pe...@qq.com> 于2020年5月27日周三 上午10:55写道：
>
> > Good idea! It's can solve the exceptions where the numbers of SSH
> > connections exceeds.
> >
> >
> > We also have another problem of how to config remote server easily when
> we
> > have hundreds of tasks.
> >
> >
> > Now I use the local param to config remote server's IP,username,password.
> > But every task needs to config.
> >
> >
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "lidong dai"<dailidong66@gmail.com>;
> > 发送时间: 2020年5月26日(星期二) 晚上6:38
> > 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> >
> > 主题: Re: [Feature] Support SSH Task
> >
> >
> >
> > good idea, I think "Control the number of task's concurrent" may be have
> > some other methods, for now, we don't have this feature, but when I
> > communicate with the team of China Unicom， they provide a new solve
> > solution：
> > we can add a task concurrent page, you can set a variable value
> like
> > 'ETL_JOB' =10 and so on, then give the task type add a option for
> choosing
> > ETL_JOB or other variable, and there may be many task also choose ETL_JOB
> > variable. when these tasks begin to running at the same time, the
> > Master
> > Server will check whether a task begin to run or wait(if the running
> tasks
> > num = 10)
> >
> >
> > Best egards
> > ---------------
> > DolphinScheduler(Incubator) PPMC
> > Lidong Dai 代立冬
> > dailidong66@gmail.com
> > ---------------
> >
> >
> > 裴龙武 <peilongwu@qq.com> 于2020年5月26日周二 下午2:34写道：
> >
> > > There is only two ways that I can think to do about this.
> > >
> > >
> > > 1. Simple SSH connect pool
> > > 2. Control the number of task's concurrent
> > >
> > >
> > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> > > 发送时间:&nbsp;2020年5月22日(星期五) 晚上7:29
> > > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> > >
> > > 主题:&nbsp;Re: [Feature] Support SSH Task
> > >
> > >
> > >
> > > What other friends think about this discuss
> > >
> > >
> > > --------------------
> > > DolphinScheduler(Incubator) Commtter
> > > Hemin Wen&nbsp; 温合民
> > > wenhemin@apache.org
> > > --------------------
> > >
> > >
> > > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月22日周五 上午10:20写道：
> > >
> > > &gt; Your are right, it can be solved the problem by modifying
> the
> > > &gt; number of server ssh connections.&amp;nbsp;
> > > &gt;
> > > &gt; The real reason is that I want to control the&amp;nbsp;
> > number of SSH
> > > &gt; connections and reuse the SSH connection, Because the
> remote
> > server is
> > > &gt; managed by other team.
> > > &gt;
> > > &gt; Otherwise, I think that we can also control the number of
> > parallel
> > > tasks
> > > &gt; in a workflow.
> > > &gt;
> > > &gt;
> > > &gt; It is a good idea that we can use the encapsulated ssh
> > connection to
> > > get
> > > &gt; clear
> > > &gt; exception information&amp;nbsp;
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt;
> > ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > > &gt; 发件人:&amp;nbsp;"Hemin Wen"<wenhemin@apache.org
> > &amp;gt;;
> > > &gt; 发送时间:&amp;nbsp;2020年5月21日(星期四) 中午11:07
> > > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> > &amp;gt;;
> > > &gt;
> > > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt; Thanks.
> > > &gt;
> > > &gt; About ssh connections pool,
> > > &gt; 1. If the connection is not enough, Can it be solved by
> > modifying the
> > > &gt; number of server ssh connections.
> > > &gt; 2. I think we have to look at whether a server's parallel
> > ssh tasks
> > > will
> > > &gt; reach the bottleneck of server connections.
> > > &gt;
> > > &gt; I think it is better to use the encapsulated ssh connection
> > to get
> > > clear
> > > &gt; exception information.
> > > &gt;
> > > &gt; --------------------
> > > &gt; DolphinScheduler(Incubator) Commtter
> > > &gt; Hemin Wen&amp;nbsp; 温合民
> > > &gt; wenhemin@apache.org
> > > &gt; --------------------
> > > &gt;
> > > &gt;
> > > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午11:29写道：
> > > &gt;
> > > &gt; &amp;gt; Thanks for your advise and suggestions. It may
> > has some
> > > exceptions if
> > > &gt; I
> > > &gt; &amp;gt; implement SSH Task as you said. Now, let me
> > explain my ideas.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; e.g
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; My workflow DAG has 100 or more task, it's all
> > ssh task. I
> > > &gt; have&amp;amp;nbsp; two
> > > &gt; &amp;gt; questions.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 1、We must limit SSH connection and reuse the
> > SSH connections,
> > > &gt; otherwise
> > > &gt; &amp;gt; the remote server will be refused.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 2、If we config all of these task, we have a
> lot
> > of repeat
> > > work to do.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; By the way, the way I'm currently verifying is
> > I use local
> > > params to
> > > &gt; &amp;gt; config the remote server's host,user,password
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > >
> >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<
> > dailidong66@gmail.com
> > > &amp;amp;gt;;
> > > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<
> > dev@dolphinscheduler.apache.org
> > > &amp;amp;gt;;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH
> > Task
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; hi, I see you will Jsch, does his license
> > compitable with
> > > apache v2
> > > &gt; &amp;gt; license, I see the following words "
> > > &gt; &amp;gt; JSch 0.0.* was released under the GNU LGPL
> > license. Later,
> > > we have
> > > &gt; switched
> > > &gt; &amp;gt; over to a BSD-style license.
> > > &gt; &amp;gt; " on
> > https://github.com/is/jsch/blob/master/LICENSE.txt .
> > > you know,
> > > &gt; LGPL
> > > &gt; &amp;gt; license not compitable with apache license.
> but
> > BSD is ok ,
> > > so the
> > > &gt; version
> > > &gt; &amp;gt; of Jsch is key point
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; second, Connection Pool is heavy resource,
> > Please use
> > > carefully
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; and I'm a little confused, why there must be
> > add a new
> > > &gt; Table,&amp;amp;nbsp; other
> > > &gt; &amp;gt; task
> > > &gt; &amp;gt; types also no seperate
> table,&amp;amp;nbsp;
> > could you
> > > consider task_type
> > > &gt; and
> > > &gt; &amp;gt; task_json
> > > &gt; &amp;gt; in t_ds_task_instance ?
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Best Regards
> > > &gt; &amp;gt; ---------------
> > > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> > > &gt; &amp;gt; Lidong Dai 代立冬
> > > &gt; &amp;gt; dailidong66@gmail.com
> > > &gt; &amp;gt; ---------------
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt;
> > 于2020年5月20日周三 下午9:57写道：
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; I want to describe my idea.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; First, we must design a host
> > table that's named
> > > &gt; t_ds_ssh_host, e.g
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; - id
> > > &gt; &amp;gt; &amp;amp;gt; - name
> > > &gt; &amp;gt; &amp;amp;gt; - ip / host
> > > &gt; &amp;gt; &amp;amp;gt; - user
> > > &gt; &amp;gt; &amp;amp;gt; - password
> > > &gt; &amp;gt; &amp;amp;gt; - max_connection
> > > &gt; &amp;gt; &amp;amp;gt; - create_time
> > > &gt; &amp;gt; &amp;amp;gt; - update_time
> > > &gt; &amp;gt; &amp;amp;gt; - user_id
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Second, Shell can execute in
> > worker server or
> > > remote host.
> > > &gt; We can
> > > &gt; &amp;gt; choose a
> > > &gt; &amp;gt; &amp;amp;gt; host in host's list.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Third, when workflow running,
> > we must maintain
> > > the remote
> > > &gt; host's
> > > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Finally, when workflow
> finish,
> > we must release
> > > the
> > > &gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Why do we have to maintain a
> > connection pool.
> > > Because it is
> > > &gt; very easy
> > > &gt; &amp;gt; to
> > > &gt; &amp;gt; &amp;amp;gt; get exceptions when it's has
> > many SSH task.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > 发件人:&amp;amp;amp;nbsp;"lidong dai"<
> > > dailidong66@gmail.com
> > > &gt; &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > > &gt; &amp;gt; &amp;amp;gt;
> > 收件人:&amp;amp;amp;nbsp;"dev"<
> > > dev@dolphinscheduler.apache.org
> > > &gt; &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re:
> > [Feature] Support SSH
> > > Task
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; glad to hear that you will
> > implement this
> > > feature
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Best Regards
> > > &gt; &amp;gt; &amp;amp;gt; ---------------
> > > &gt; &amp;gt; &amp;amp;gt; DolphinScheduler(Incubator)
> > PPMC
> > > &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬
> > > &gt; &amp;gt; &amp;amp;gt; dailidong66@gmail.com
> > > &gt; &amp;gt; &amp;amp;gt; ---------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 裴龙武 <peilongwu@qq.com
> > &amp;amp;amp;gt;
> > > 于2020年5月20日周三 下午3:47写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; My code
> > is not perfect yet. I
> > > will write a
> > > &gt; detailed design
> > > &gt; &amp;gt; document.
> > > &gt; &amp;gt; &amp;amp;gt; Then
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I will
> > realize this feature
> > > about our
> > > &gt; discussion result.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 发件人:&amp;amp;amp;amp;nbsp;"wenhemin"<
> > > &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 发送时间:&amp;amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 收件人:&amp;amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;gt;;"dev"<
> > > &gt; &amp;gt; &amp;amp;gt;
> > dev@dolphinscheduler.apache.org
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 主题:&amp;amp;amp;amp;nbsp;Re:
> > > [Feature] Support SSH
> > > &gt; Task and Support
> > > &gt; &amp;gt; dummy task
> > > &gt; &amp;gt; &amp;amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Thanks
> > for writing detailed
> > > documentation. I
> > > &gt; think this is
> > > &gt; &amp;gt; also a
> > > &gt; &amp;gt; &amp;amp;gt; missing
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; feature
> > of DS.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About
> the
> > extension point:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 1.Can
> ssh
> > tasks be merged into
> > > shell tasks.
> > > &gt; Essentially,
> > > &gt; &amp;gt; they all
> > > &gt; &amp;gt; &amp;amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; shell
> > commands.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 2.About
> > dummy task, DS has the
> > > function of
> > > &gt; disable nodes, I
> > > &gt; &amp;gt; do n’t
> > > &gt; &amp;gt; &amp;amp;gt; know if
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> > requirement is met.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; The
> > script from AirFlow to
> > > Dolphin is great.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; 在
> > > 2020年5月18日，09:28，裴龙武 <
> > > &gt; peilongwu@qq.com&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; OK, 3Q!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; First, I
> > > will ensure that open
> > > &gt; source can use.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; Second, I
> > > think we must
> > > &gt; discuss deeply. I write
> > > &gt; &amp;gt; a more
> > > &gt; &amp;gt; &amp;amp;gt; detailed
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> document.
> > You can check the
> > > attachment. I also
> > > &gt; send the
> > > &gt; &amp;gt; document to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> DaiLidong.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > Third,&amp;amp;amp;amp;nbsp; I'll
> > > &gt; give you the error of
> > > &gt; &amp;gt; not using SSH
> > > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > ------------------ 原始邮件
> > > &gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; 发件人:
> > > "wenhemin"<
> > > &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; 发送时间:
> > > 2020年5月14日(星期四) 晚上7:26
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; 收件人:
> > > "裴龙武"<peilongwu@qq.com
> > > &gt; &amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; 主题: Re:
> > > [Feature] Support SSH
> > > &gt; Task and Support
> > > &gt; &amp;gt; dummy task
> > > &gt; &amp;gt; &amp;amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; Great!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; I think,
> > > Can ssh tasks be
> > > &gt; merged into shell
> > > &gt; &amp;gt; tasks,&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script
> > locally or remotely,
> > > Configure on the
> > > &gt; front end.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; About ssh
> > > connect pool, I did
> > > &gt; not find it
> > > &gt; &amp;gt; necessary to use
> > > &gt; &amp;gt; &amp;amp;gt; the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt; BTW, Look
> > > at the code to
> > > &gt; introduce additional
> > > &gt; &amp;gt; jar packages,
> > > &gt; &amp;gt; &amp;amp;gt; You also
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; need to
> > ensure that open
> > > source can use the
> > > &gt; license of this
> > > &gt; &amp;gt; jar
> > > &gt; &amp;gt; &amp;amp;gt; package.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在
> > > &gt; 2020年5月14日，16:20，裴龙武 <
> > > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1. The
> > > &gt; priority between these tasks
> > > &gt; &amp;gt; is also depended
> > > &gt; &amp;gt; &amp;amp;gt; on the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dolphin
> > DAG define. When the
> > > front task is not
> > > &gt; finished, it
> > > &gt; &amp;gt; not
> > > &gt; &amp;gt; &amp;amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; next
> task.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2. I extend
> > > &gt; ssh task. I also use
> > > &gt; &amp;gt; local params to
> > > &gt; &amp;gt; &amp;amp;gt; config ssh host,
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user and
> > password.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; E.g:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; public static
> > > &gt; AbstractTask
> > > &gt; &amp;gt; &amp;amp;gt; newTask(TaskExecutionContext
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > taskExecutionContext, Logger
> > > logger)
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; throws
> > > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > Boolean
> > > &gt; &amp;gt; enable =
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > if
> > > &gt; &amp;gt; (enable != null
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;
> > > enable == false ) {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > DummyTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > switch
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case SHELL:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > ShellTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case
> > > &gt; &amp;gt; &amp;amp;gt; PROCEDURE:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > ProcedureTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case SQL:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > SqlTask(taskExecutionContext,
> > > logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case MR:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > MapReduceTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case SPARK:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > SparkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case FLINK:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > FlinkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case PYTHON:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > PythonTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case HTTP:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > HttpTask(taskExecutionContext,
> > > logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case DATAX:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > DataxTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case SQOOP:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > SqoopTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; case SSH:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > SSHTask(taskExecutionContext,
> > > logger);
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; default:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; logger.error("unsupport task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; type:
> {}",
> > > taskExecutionContext.getTaskType());
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; throw new
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > IllegalArgumentException("not
> > > support task
> > > &gt; type");
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; }
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3. I am not
> > > &gt; sure that it supports
> > > &gt; &amp;gt; window or not.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; ------------------ 原始邮件
> > > &gt; &amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发件人:
> > > &gt; "wenhemin"<whm_777@163.com
> > > &gt; &amp;gt; <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发送时间:
> > > &gt; 2020年5月14日(星期四) 下午3:46
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 收件人: "裴龙武"<
> > > &gt; peilongwu@qq.com
> > > &gt; &amp;gt; <mailto:peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 主题: Re:
> > > &gt; [Feature] Support SSH Task
> > > &gt; &amp;gt; and Support dummy
> > > &gt; &amp;gt; &amp;amp;gt; task like
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; Sorry, My
> > > &gt; previous description is
> > > &gt; &amp;gt; not very clear.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I want to ask
> > > &gt; some questions:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1.How to
> > > &gt; control the priority
> > > &gt; &amp;gt; between ssh tasks?
> > > &gt; &amp;gt; &amp;amp;gt; There may be
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; some ssh
> > tasks that have been
> > > waiting for
> > > &gt; execution.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2.I understand
> > > &gt; what you want to
> > > &gt; &amp;gt; solve is the problem
> > > &gt; &amp;gt; &amp;amp;gt; of executing
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; remote
> > ssh scripts in batches.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > So, not
> > > &gt; &amp;gt; sure how to use this
> > > &gt; &amp;gt; &amp;amp;gt; function.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3.I don't know
> > > &gt; if this supports
> > > &gt; &amp;gt; windows system.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 在
> > > 2020年5月13日，20:56，裴龙武 <
> > > &gt; &amp;gt; peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> I
> > use
> > > spin lock. Here
> > > &gt; &amp;gt; is my code. Of course
> > > &gt; &amp;gt; &amp;amp;gt; , it's not
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perfect.
> > I just do a test. To
> > > my surprise, it
> > > &gt; is the result
> > > &gt; &amp;gt; of the
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> execution
> > is the same as the
> > > AirFlow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 相同的结果。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > ------------------ 原始邮件
> > > &gt; &amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 发件人:
> > > "whm_777"<
> > > &gt; &amp;gt; whm_777@163.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 发送时间:
> > > 2020年5月13日(星期三)
> > > &gt; &amp;gt; 晚上7:21
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 收件人:
> > > "裴龙武"<
> > > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 主题: Re:
> > > [Feature]
> > > &gt; &amp;gt; Support SSH Task and
> > > &gt; &amp;gt; &amp;amp;gt; Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > You can
> > > modify the
> > > &gt; &amp;gt; maximum number of linux
> > > &gt; &amp;gt; &amp;amp;gt; ssh connections.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> If
> > use
> > > ssh connection
> > > &gt; &amp;gt; pool, How to control
> > > &gt; &amp;gt; &amp;amp;gt; the priority of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh?
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > 在
> > > &gt; &amp;gt; 2020年5月13日，18:01，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > First 3Q，
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; I&amp;amp;amp;amp;nbsp; use more than 100
> > task
> > > &gt; &amp;gt; &amp;amp;gt; node. But SSH connections
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are
> > limited.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; <330BE502@F7F80E73.76C5BB5E.jpg
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > 发件人:
> > > &gt; &amp;gt; "whm_777"<whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; <mailto:whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > 发送时间:
> > > &gt; &amp;gt; 2020年5月13日(星期三) 下午5:50
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > 收件人: "裴龙武"<
> > > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > 主题: Re:
> > > &gt; &amp;gt; [Feature] Support SSH Task
> > > &gt; &amp;gt; &amp;amp;gt; and Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > E.g.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; rtn_code=`ssh -o
> > > &gt; &amp;gt; &amp;amp;gt; ServerAliveInterval=60 -p
> xxxx
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > root@xxx.xxx.xxx.xxx <mailto:
> > > &gt; root@xxx.xxx.xxx.xxx&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; ‘shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > command&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;amp;amp;amp;gt;/dev/null
> > > &gt; &amp;gt;
> > 2&amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;1; echo $?'`
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > if [
> > > &gt; &amp;gt; "$rtn_code" -eq 0 ]; then
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; echo "成功"
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; exit 0
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > else
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; echo "失败"
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; exit 1
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > fi
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > Batch shell
> > > &gt; &amp;gt; command is not supported.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > Multiple
> > > &gt; &amp;gt; servers can be split into
> > > &gt; &amp;gt; &amp;amp;gt; multiple task nodes.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; Could you give me a
> > > &gt; &amp;gt; &amp;amp;gt; example，3Q. 能否给我一个例子，谢谢！
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; By the way, I have more than
> > > &gt; &amp;gt; &amp;amp;gt; 100 tasks in one DAG.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; These
> > tasks connect two other
> > > server to
> > > &gt; execute. So SSH
> > > &gt; &amp;gt; tasks must
> > > &gt; &amp;gt; &amp;amp;gt; have
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; pool to
> > manager. Now I use
> > > JSch and realize a
> > > &gt; simple pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > > &gt; &amp;gt; &amp;amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH
> > 任务进行连接时，必须使用连接池进行管理。当前我使用
> > > JSch，并实现了一个简单的连接池。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; ------------------ 原始邮件
> > > &gt; &amp;gt; &amp;amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 发件人: "wenhemin"<
> > > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > whm_777@163.com
> > > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 收件人: "dev"<
> > > &gt; &amp;gt; &amp;amp;gt;
> > dev@dolphinscheduler.apache.org <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > dev@dolphinscheduler.apache.org
> > > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; 主题: Re: [Feature] Support
> > > &gt; &amp;gt; &amp;amp;gt; SSH Task and Support dummy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task
> like
> > airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; The shell node is supports
> > > &gt; &amp;gt; &amp;amp;gt; remote calling, and get
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the
> > remote command result code.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 在
> > > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，15:16，裴龙武 <
> > peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto:
> > peilongwu@qq.com
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dear ALL：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support Linux SSH
> > > &gt; &amp;gt; &amp;amp;gt; Task 支持 Linux SSH 任务
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器
> > Shell 脚本，Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 脚本是保存在业务服务器的固定目录。当 Worker
> > > &gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; &amp;amp;gt; project, the workflow's tasks
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; want to
> > execute shell scripts
> > > where are in
> > > &gt; different server's
> > > &gt; &amp;gt; &amp;amp;gt; different
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > directory. When worker execute
> > > these shell
> > > &gt; scripts, it must
> > > &gt; &amp;gt; use the
> > > &gt; &amp;gt; &amp;amp;gt; same
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user to
> > login these server.
> > > Also, the worker
> > > &gt; can get the
> > > &gt; &amp;gt; executing
> > > &gt; &amp;gt; &amp;amp;gt; state of
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; these
> > server. We can config
> > > these server 's
> > > &gt; host,user and
> > > &gt; &amp;gt; password.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; SSH Task is very
> > > &gt; &amp;gt; &amp;amp;gt; useful for most user SSH
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 任务对大多数用户是非常有用的
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > > &gt; Worker
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; In
> > > &gt; &amp;gt; &amp;amp;gt; dolphinscheduler, the most
> > executing tasks
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are in
> > different servers who
> > > are not workers.
> > > &gt; These servers
> > > &gt; &amp;gt; also have
> > > &gt; &amp;gt; &amp;amp;gt; their
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> different
> > fixed services. We
> > > just have to pass
> > > &gt; different
> > > &gt; &amp;gt; parameters to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; schedule
> > these shell scripts
> > > to execute.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; &amp;amp;gt; to execute ssh script Python
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 有固定的工具包，可执行这些SSH Shell 脚本
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python
> > > &gt; &amp;gt; &amp;amp;gt; 有一个可执行远程服务器SSH
> > Shell脚本的模块，其名字为：paramiko。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; &amp;amp;gt; that can execute SSH Shell
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script.
> > It's paramiko.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Others 其他内容
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; I found this
> > > &gt; &amp;gt; &amp;amp;gt; described in previous
> feature,
> > but
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; it was
> > relatively simple.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Feature URL
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 另外，我通过 Shell Task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; In addition, it is
> > > &gt; &amp;gt; &amp;amp;gt; very inconvenient for me to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perform
> > remote tasks through
> > > Shell Task. Here
> > > &gt; is my script.
> > > &gt; &amp;gt; I don't
> > > &gt; &amp;gt; &amp;amp;gt; know if
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; there's
> a
> > better way.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; sshpass -p
> > > &gt; &amp;gt; &amp;amp;gt; 'password' ssh user@host
> echo
> > 'ssh
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; success'
> > echo 'Hello World'
> > > &gt; -&amp;amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > /home/dolphinscheduler/test/hello.txt
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo
> 'end'
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; like airflow 支持像 Airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 中的虚拟任务
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > > &gt; &amp;gt; &amp;amp;gt; DAG 文件，DAG
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > > &gt; &amp;gt; &amp;amp;gt; Dummy
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> > Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; &amp;amp;gt; project, it has a productized
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG
> file.
> > The file contains
> > > different modules,
> > > &gt; some of which
> > > &gt; &amp;gt; are
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > interdependent and some of
> > > which are not. When
> > > &gt; customers
> > > &gt; &amp;gt; purchase
> > > &gt; &amp;gt; &amp;amp;gt; different
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; modules,
> > we need to set some
> > > tasks as dummy
> > > &gt; tasks, which
> > > &gt; &amp;gt; some modules
> > > &gt; &amp;gt; &amp;amp;gt; are
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; not
> > purchased and the
> > > purchased module is not
> > > &gt; dependent.
> > > &gt; &amp;gt; Because of
> > > &gt; &amp;gt; &amp;amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setting,
> > these dummy tasks are
> > > actually not
> > > &gt; executed. The
> > > &gt; &amp;gt; benefits of
> > > &gt; &amp;gt; &amp;amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setup
> are
> > product unity and
> > > diagram integrity.
> > > &gt; In airflow,
> > > &gt; &amp;gt; these task
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute
> > by dummy operator.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; ** Realize 实现方式**
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task
> > > &gt; &amp;gt; &amp;amp;gt;
> > 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy
> > 时，实际的任务不执行，执行 Dummy Task。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task is easy
> > > &gt; &amp;gt; &amp;amp;gt; to realize, but it need to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; use with
> > other different
> > > tasks. When the task's
> > > &gt; executed
> > > &gt; &amp;gt; type is set
> > > &gt; &amp;gt; &amp;amp;gt; to
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy
> > type, the task are
> > > executed as a dummy
> > > &gt; task and the
> > > &gt; &amp;gt; real task
> > > &gt; &amp;gt; &amp;amp;gt; is not
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> executed.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;amp;amp;gt; By the way，I
> > > &gt; &amp;gt; &amp;amp;gt; already realize these
> > > two&amp;amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; features
> > in my fork
> > > &gt; branch.&amp;amp;amp;amp;amp;nbsp;Whether the
> > > &gt; &amp;gt; follow-up
> > > &gt; &amp;gt; &amp;amp;gt; release can be
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> supported
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > >
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> <SSHClient.java&amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;
> > > &gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;gt;
>
Re: [Feature] Support SSH Task
Posted by Hemin Wen <we...@apache.org>.
Is it possible to extend the data source and increase the ssh type.
I look at the data structure of DB is ok.
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
裴龙武 <pe...@qq.com> 于2020年5月27日周三 上午10:55写道：
> Good idea! It's can solve the exceptions where the numbers of SSH
> connections exceeds.
>
>
> We also have another problem of how to config remote server easily when we
> have hundreds of tasks.
>
>
> Now I use the local param to config remote server's IP,username,password.
> But every task needs to config.
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "lidong dai"<dailidong66@gmail.com>;
> 发送时间: 2020年5月26日(星期二) 晚上6:38
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> good idea, I think "Control the number of task's concurrent" may be have
> some other methods, for now, we don't have this feature, but when I
> communicate with the team of China Unicom， they provide a new solve
> solution：
> we can add a task concurrent page, you can set a variable value like
> 'ETL_JOB' =10 and so on, then give the task type add a option for choosing
> ETL_JOB or other variable, and there may be many task also choose ETL_JOB
> variable. when these tasks begin to running at the same time, the
> Master
> Server will check whether a task begin to run or wait(if the running tasks
> num = 10)
>
>
> Best egards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月26日周二 下午2:34写道：
>
> > There is only two ways that I can think to do about this.
> >
> >
> > 1. Simple SSH connect pool
> > 2. Control the number of task's concurrent
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> > 发送时间:&nbsp;2020年5月22日(星期五) 晚上7:29
> > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task
> >
> >
> >
> > What other friends think about this discuss
> >
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen&nbsp; 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月22日周五 上午10:20写道：
> >
> > &gt; Your are right, it can be solved the problem by modifying the
> > &gt; number of server ssh connections.&amp;nbsp;
> > &gt;
> > &gt; The real reason is that I want to control the&amp;nbsp;
> number of SSH
> > &gt; connections and reuse the SSH connection, Because the remote
> server is
> > &gt; managed by other team.
> > &gt;
> > &gt; Otherwise, I think that we can also control the number of
> parallel
> > tasks
> > &gt; in a workflow.
> > &gt;
> > &gt;
> > &gt; It is a good idea that we can use the encapsulated ssh
> connection to
> > get
> > &gt; clear
> > &gt; exception information&amp;nbsp;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:&amp;nbsp;"Hemin Wen"<wenhemin@apache.org
> &amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2020年5月21日(星期四) 中午11:07
> > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;gt;;
> > &gt;
> > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > &gt;
> > &gt;
> > &gt;
> > &gt; Thanks.
> > &gt;
> > &gt; About ssh connections pool,
> > &gt; 1. If the connection is not enough, Can it be solved by
> modifying the
> > &gt; number of server ssh connections.
> > &gt; 2. I think we have to look at whether a server's parallel
> ssh tasks
> > will
> > &gt; reach the bottleneck of server connections.
> > &gt;
> > &gt; I think it is better to use the encapsulated ssh connection
> to get
> > clear
> > &gt; exception information.
> > &gt;
> > &gt; --------------------
> > &gt; DolphinScheduler(Incubator) Commtter
> > &gt; Hemin Wen&amp;nbsp; 温合民
> > &gt; wenhemin@apache.org
> > &gt; --------------------
> > &gt;
> > &gt;
> > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午11:29写道：
> > &gt;
> > &gt; &amp;gt; Thanks for your advise and suggestions. It may
> has some
> > exceptions if
> > &gt; I
> > &gt; &amp;gt; implement SSH Task as you said. Now, let me
> explain my ideas.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; e.g
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; My workflow DAG has 100 or more task, it's all
> ssh task. I
> > &gt; have&amp;amp;nbsp; two
> > &gt; &amp;gt; questions.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 1、We must limit SSH connection and reuse the
> SSH connections,
> > &gt; otherwise
> > &gt; &amp;gt; the remote server will be refused.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 2、If we config all of these task, we have a lot
> of repeat
> > work to do.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; By the way, the way I'm currently verifying is
> I use local
> > params to
> > &gt; &amp;gt; config the remote server's host,user,password
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<
> dailidong66@gmail.com
> > &amp;amp;gt;;
> > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<
> dev@dolphinscheduler.apache.org
> > &amp;amp;gt;;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH
> Task
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; hi, I see you will Jsch, does his license
> compitable with
> > apache v2
> > &gt; &amp;gt; license, I see the following words "
> > &gt; &amp;gt; JSch 0.0.* was released under the GNU LGPL
> license. Later,
> > we have
> > &gt; switched
> > &gt; &amp;gt; over to a BSD-style license.
> > &gt; &amp;gt; " on
> https://github.com/is/jsch/blob/master/LICENSE.txt .
> > you know,
> > &gt; LGPL
> > &gt; &amp;gt; license not compitable with apache license. but
> BSD is ok ,
> > so the
> > &gt; version
> > &gt; &amp;gt; of Jsch is key point
> > &gt; &amp;gt;
> > &gt; &amp;gt; second, Connection Pool is heavy resource,
> Please use
> > carefully
> > &gt; &amp;gt;
> > &gt; &amp;gt; and I'm a little confused, why there must be
> add a new
> > &gt; Table,&amp;amp;nbsp; other
> > &gt; &amp;gt; task
> > &gt; &amp;gt; types also no seperate table,&amp;amp;nbsp;
> could you
> > consider task_type
> > &gt; and
> > &gt; &amp;gt; task_json
> > &gt; &amp;gt; in t_ds_task_instance ?
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Best Regards
> > &gt; &amp;gt; ---------------
> > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> > &gt; &amp;gt; Lidong Dai 代立冬
> > &gt; &amp;gt; dailidong66@gmail.com
> > &gt; &amp;gt; ---------------
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt;
> 于2020年5月20日周三 下午9:57写道：
> > &gt; &amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; I want to describe my idea.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; First, we must design a host
> table that's named
> > &gt; t_ds_ssh_host, e.g
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; - id
> > &gt; &amp;gt; &amp;amp;gt; - name
> > &gt; &amp;gt; &amp;amp;gt; - ip / host
> > &gt; &amp;gt; &amp;amp;gt; - user
> > &gt; &amp;gt; &amp;amp;gt; - password
> > &gt; &amp;gt; &amp;amp;gt; - max_connection
> > &gt; &amp;gt; &amp;amp;gt; - create_time
> > &gt; &amp;gt; &amp;amp;gt; - update_time
> > &gt; &amp;gt; &amp;amp;gt; - user_id
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Second, Shell can execute in
> worker server or
> > remote host.
> > &gt; We can
> > &gt; &amp;gt; choose a
> > &gt; &amp;gt; &amp;amp;gt; host in host's list.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Third, when workflow running,
> we must maintain
> > the remote
> > &gt; host's
> > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Finally, when workflow finish,
> we must release
> > the
> > &gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Why do we have to maintain a
> connection pool.
> > Because it is
> > &gt; very easy
> > &gt; &amp;gt; to
> > &gt; &amp;gt; &amp;amp;gt; get exceptions when it's has
> many SSH task.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt;
> 发件人:&amp;amp;amp;nbsp;"lidong dai"<
> > dailidong66@gmail.com
> > &gt; &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > &gt; &amp;gt; &amp;amp;gt;
> 收件人:&amp;amp;amp;nbsp;"dev"<
> > dev@dolphinscheduler.apache.org
> > &gt; &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re:
> [Feature] Support SSH
> > Task
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; glad to hear that you will
> implement this
> > feature
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Best Regards
> > &gt; &amp;gt; &amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt; DolphinScheduler(Incubator)
> PPMC
> > &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬
> > &gt; &amp;gt; &amp;amp;gt; dailidong66@gmail.com
> > &gt; &amp;gt; &amp;amp;gt; ---------------
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 裴龙武 <peilongwu@qq.com
> &amp;amp;amp;gt;
> > 于2020年5月20日周三 下午3:47写道：
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; My code
> is not perfect yet. I
> > will write a
> > &gt; detailed design
> > &gt; &amp;gt; document.
> > &gt; &amp;gt; &amp;amp;gt; Then
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I will
> realize this feature
> > about our
> > &gt; discussion result.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 发件人:&amp;amp;amp;amp;nbsp;"wenhemin"<
> > &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 发送时间:&amp;amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 收件人:&amp;amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;gt;;"dev"<
> > &gt; &amp;gt; &amp;amp;gt;
> dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 主题:&amp;amp;amp;amp;nbsp;Re:
> > [Feature] Support SSH
> > &gt; Task and Support
> > &gt; &amp;gt; dummy task
> > &gt; &amp;gt; &amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Thanks
> for writing detailed
> > documentation. I
> > &gt; think this is
> > &gt; &amp;gt; also a
> > &gt; &amp;gt; &amp;amp;gt; missing
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; feature
> of DS.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About the
> extension point:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 1.Can ssh
> tasks be merged into
> > shell tasks.
> > &gt; Essentially,
> > &gt; &amp;gt; they all
> > &gt; &amp;gt; &amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; shell
> commands.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 2.About
> dummy task, DS has the
> > function of
> > &gt; disable nodes, I
> > &gt; &amp;gt; do n’t
> > &gt; &amp;gt; &amp;amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this
> requirement is met.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; The
> script from AirFlow to
> > Dolphin is great.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 在
> > 2020年5月18日，09:28，裴龙武 <
> > &gt; peilongwu@qq.com&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; OK, 3Q!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; First, I
> > will ensure that open
> > &gt; source can use.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; Second, I
> > think we must
> > &gt; discuss deeply. I write
> > &gt; &amp;gt; a more
> > &gt; &amp;gt; &amp;amp;gt; detailed
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; document.
> You can check the
> > attachment. I also
> > &gt; send the
> > &gt; &amp;gt; document to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DaiLidong.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > Third,&amp;amp;amp;amp;nbsp; I'll
> > &gt; give you the error of
> > &gt; &amp;gt; not using SSH
> > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > ------------------ 原始邮件
> > &gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 发件人:
> > "wenhemin"<
> > &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 发送时间:
> > 2020年5月14日(星期四) 晚上7:26
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 收件人:
> > "裴龙武"<peilongwu@qq.com
> > &gt; &amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; 主题: Re:
> > [Feature] Support SSH
> > &gt; Task and Support
> > &gt; &amp;gt; dummy task
> > &gt; &amp;gt; &amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; Great!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; I think,
> > Can ssh tasks be
> > &gt; merged into shell
> > &gt; &amp;gt; tasks,&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script
> locally or remotely,
> > Configure on the
> > &gt; front end.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; About ssh
> > connect pool, I did
> > &gt; not find it
> > &gt; &amp;gt; necessary to use
> > &gt; &amp;gt; &amp;amp;gt; the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt; BTW, Look
> > at the code to
> > &gt; introduce additional
> > &gt; &amp;gt; jar packages,
> > &gt; &amp;gt; &amp;amp;gt; You also
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; need to
> ensure that open
> > source can use the
> > &gt; license of this
> > &gt; &amp;gt; jar
> > &gt; &amp;gt; &amp;amp;gt; package.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在
> > &gt; 2020年5月14日，16:20，裴龙武 <
> > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1. The
> > &gt; priority between these tasks
> > &gt; &amp;gt; is also depended
> > &gt; &amp;gt; &amp;amp;gt; on the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dolphin
> DAG define. When the
> > front task is not
> > &gt; finished, it
> > &gt; &amp;gt; not
> > &gt; &amp;gt; &amp;amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; next task.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2. I extend
> > &gt; ssh task. I also use
> > &gt; &amp;gt; local params to
> > &gt; &amp;gt; &amp;amp;gt; config ssh host,
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user and
> password.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; E.g:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; public static
> > &gt; AbstractTask
> > &gt; &amp;gt; &amp;amp;gt; newTask(TaskExecutionContext
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> taskExecutionContext, Logger
> > logger)
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; throws
> > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > Boolean
> > &gt; &amp;gt; enable =
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > if
> > &gt; &amp;gt; (enable != null
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;
> > enable == false ) {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > DummyTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > switch
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case SHELL:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > ShellTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case
> > &gt; &amp;gt; &amp;amp;gt; PROCEDURE:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > ProcedureTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case SQL:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> SqlTask(taskExecutionContext,
> > logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case MR:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > MapReduceTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case SPARK:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > SparkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case FLINK:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > FlinkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case PYTHON:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > PythonTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case HTTP:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> HttpTask(taskExecutionContext,
> > logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case DATAX:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > DataxTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case SQOOP:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > SqoopTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; case SSH:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> SSHTask(taskExecutionContext,
> > logger);
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; default:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; logger.error("unsupport task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; type: {}",
> > taskExecutionContext.getTaskType());
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; throw new
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> IllegalArgumentException("not
> > support task
> > &gt; type");
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; }
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3. I am not
> > &gt; sure that it supports
> > &gt; &amp;gt; window or not.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; ------------------ 原始邮件
> > &gt; &amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发件人:
> > &gt; "wenhemin"<whm_777@163.com
> > &gt; &amp;gt; <mailto:
> > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发送时间:
> > &gt; 2020年5月14日(星期四) 下午3:46
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 收件人: "裴龙武"<
> > &gt; peilongwu@qq.com
> > &gt; &amp;gt; <mailto:peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 主题: Re:
> > &gt; [Feature] Support SSH Task
> > &gt; &amp;gt; and Support dummy
> > &gt; &amp;gt; &amp;amp;gt; task like
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; Sorry, My
> > &gt; previous description is
> > &gt; &amp;gt; not very clear.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I want to ask
> > &gt; some questions:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1.How to
> > &gt; control the priority
> > &gt; &amp;gt; between ssh tasks?
> > &gt; &amp;gt; &amp;amp;gt; There may be
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; some ssh
> tasks that have been
> > waiting for
> > &gt; execution.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2.I understand
> > &gt; what you want to
> > &gt; &amp;gt; solve is the problem
> > &gt; &amp;gt; &amp;amp;gt; of executing
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; remote
> ssh scripts in batches.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > So, not
> > &gt; &amp;gt; sure how to use this
> > &gt; &amp;gt; &amp;amp;gt; function.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3.I don't know
> > &gt; if this supports
> > &gt; &amp;gt; windows system.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在
> > 2020年5月13日，20:56，裴龙武 <
> > &gt; &amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I
> use
> > spin lock. Here
> > &gt; &amp;gt; is my code. Of course
> > &gt; &amp;gt; &amp;amp;gt; , it's not
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perfect.
> I just do a test. To
> > my surprise, it
> > &gt; is the result
> > &gt; &amp;gt; of the
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execution
> is the same as the
> > AirFlow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 相同的结果。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > ------------------ 原始邮件
> > &gt; &amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 发件人:
> > "whm_777"<
> > &gt; &amp;gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 发送时间:
> > 2020年5月13日(星期三)
> > &gt; &amp;gt; 晚上7:21
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 收件人:
> > "裴龙武"<
> > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 主题: Re:
> > [Feature]
> > &gt; &amp;gt; Support SSH Task and
> > &gt; &amp;gt; &amp;amp;gt; Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> You can
> > modify the
> > &gt; &amp;gt; maximum number of linux
> > &gt; &amp;gt; &amp;amp;gt; ssh connections.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; If
> use
> > ssh connection
> > &gt; &amp;gt; pool, How to control
> > &gt; &amp;gt; &amp;amp;gt; the priority of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh?
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 在
> > &gt; &amp;gt; 2020年5月13日，18:01，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > First 3Q，
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; I&amp;amp;amp;amp;nbsp; use more than 100
> task
> > &gt; &amp;gt; &amp;amp;gt; node. But SSH connections
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are
> limited.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; <330BE502@F7F80E73.76C5BB5E.jpg
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 发件人:
> > &gt; &amp;gt; "whm_777"<whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; <mailto:whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 发送时间:
> > &gt; &amp;gt; 2020年5月13日(星期三) 下午5:50
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 收件人: "裴龙武"<
> > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > 主题: Re:
> > &gt; &amp;gt; [Feature] Support SSH Task
> > &gt; &amp;gt; &amp;amp;gt; and Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like
> airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > E.g.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; rtn_code=`ssh -o
> > &gt; &amp;gt; &amp;amp;gt; ServerAliveInterval=60 -p xxxx
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> root@xxx.xxx.xxx.xxx <mailto:
> > &gt; root@xxx.xxx.xxx.xxx&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; ‘shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> command&amp;amp;amp;amp;nbsp;
> > &gt; &amp;amp;amp;amp;gt;/dev/null
> > &gt; &amp;gt;
> 2&amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;1; echo $?'`
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > if [
> > &gt; &amp;gt; "$rtn_code" -eq 0 ]; then
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; echo "成功"
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; exit 0
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > else
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; echo "失败"
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; exit 1
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > fi
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > Batch shell
> > &gt; &amp;gt; command is not supported.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > Multiple
> > &gt; &amp;gt; servers can be split into
> > &gt; &amp;gt; &amp;amp;gt; multiple task nodes.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; Could you give me a
> > &gt; &amp;gt; &amp;amp;gt; example，3Q. 能否给我一个例子，谢谢！
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; By the way, I have more than
> > &gt; &amp;gt; &amp;amp;gt; 100 tasks in one DAG.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; These
> tasks connect two other
> > server to
> > &gt; execute. So SSH
> > &gt; &amp;gt; tasks must
> > &gt; &amp;gt; &amp;amp;gt; have
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; pool to
> manager. Now I use
> > JSch and realize a
> > &gt; simple pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > &gt; &amp;gt; &amp;amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH
> 任务进行连接时，必须使用连接池进行管理。当前我使用
> > JSch，并实现了一个简单的连接池。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; ------------------ 原始邮件
> > &gt; &amp;gt; &amp;amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 发件人: "wenhemin"<
> > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> whm_777@163.com
> > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 收件人: "dev"<
> > &gt; &amp;gt; &amp;amp;gt;
> dev@dolphinscheduler.apache.org <mailto:
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > dev@dolphinscheduler.apache.org
> > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; 主题: Re: [Feature] Support
> > &gt; &amp;gt; &amp;amp;gt; SSH Task and Support dummy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task like
> airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; The shell node is supports
> > &gt; &amp;gt; &amp;amp;gt; remote calling, and get
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the
> remote command result code.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; 在
> > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，15:16，裴龙武 <
> peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto:
> peilongwu@qq.com
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dear ALL：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support Linux SSH
> > &gt; &amp;gt; &amp;amp;gt; Task 支持 Linux SSH 任务
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器
> Shell 脚本，Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 脚本是保存在业务服务器的固定目录。当 Worker
> > &gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; &amp;amp;gt; project, the workflow's tasks
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; want to
> execute shell scripts
> > where are in
> > &gt; different server's
> > &gt; &amp;gt; &amp;amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> directory. When worker execute
> > these shell
> > &gt; scripts, it must
> > &gt; &amp;gt; use the
> > &gt; &amp;gt; &amp;amp;gt; same
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user to
> login these server.
> > Also, the worker
> > &gt; can get the
> > &gt; &amp;gt; executing
> > &gt; &amp;gt; &amp;amp;gt; state of
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; these
> server. We can config
> > these server 's
> > &gt; host,user and
> > &gt; &amp;gt; password.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; SSH Task is very
> > &gt; &amp;gt; &amp;amp;gt; useful for most user SSH
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 任务对大多数用户是非常有用的
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > &gt; Worker
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; In
> > &gt; &amp;gt; &amp;amp;gt; dolphinscheduler, the most
> executing tasks
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are in
> different servers who
> > are not workers.
> > &gt; These servers
> > &gt; &amp;gt; also have
> > &gt; &amp;gt; &amp;amp;gt; their
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; different
> fixed services. We
> > just have to pass
> > &gt; different
> > &gt; &amp;gt; parameters to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; schedule
> these shell scripts
> > to execute.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; &amp;amp;gt; to execute ssh script Python
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python
> > &gt; &amp;gt; &amp;amp;gt; 有一个可执行远程服务器SSH
> Shell脚本的模块，其名字为：paramiko。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; &amp;amp;gt; that can execute SSH Shell
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script.
> It's paramiko.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Others 其他内容
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; I found this
> > &gt; &amp;gt; &amp;amp;gt; described in previous feature,
> but
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; it was
> relatively simple.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Feature URL
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; 另外，我通过 Shell Task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; In addition, it is
> > &gt; &amp;gt; &amp;amp;gt; very inconvenient for me to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perform
> remote tasks through
> > Shell Task. Here
> > &gt; is my script.
> > &gt; &amp;gt; I don't
> > &gt; &amp;gt; &amp;amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; there's a
> better way.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; sshpass -p
> > &gt; &amp;gt; &amp;amp;gt; 'password' ssh user@host echo
> 'ssh
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; success'
> echo 'Hello World'
> > &gt; -&amp;amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> /home/dolphinscheduler/test/hello.txt
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo 'end'
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; like airflow 支持像 Airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 中的虚拟任务
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > &gt; &amp;gt; &amp;amp;gt; DAG 文件，DAG
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt;
> > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > &gt; &amp;gt; &amp;amp;gt; Dummy
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; &amp;amp;gt; project, it has a productized
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG file.
> The file contains
> > different modules,
> > &gt; some of which
> > &gt; &amp;gt; are
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> interdependent and some of
> > which are not. When
> > &gt; customers
> > &gt; &amp;gt; purchase
> > &gt; &amp;gt; &amp;amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; modules,
> we need to set some
> > tasks as dummy
> > &gt; tasks, which
> > &gt; &amp;gt; some modules
> > &gt; &amp;gt; &amp;amp;gt; are
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; not
> purchased and the
> > purchased module is not
> > &gt; dependent.
> > &gt; &amp;gt; Because of
> > &gt; &amp;gt; &amp;amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setting,
> these dummy tasks are
> > actually not
> > &gt; executed. The
> > &gt; &amp;gt; benefits of
> > &gt; &amp;gt; &amp;amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setup are
> product unity and
> > diagram integrity.
> > &gt; In airflow,
> > &gt; &amp;gt; these task
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute
> by dummy operator.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; ** Realize 实现方式**
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task
> > &gt; &amp;gt; &amp;amp;gt;
> 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy
> 时，实际的任务不执行，执行 Dummy Task。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task is easy
> > &gt; &amp;gt; &amp;amp;gt; to realize, but it need to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; use with
> other different
> > tasks. When the task's
> > &gt; executed
> > &gt; &amp;gt; type is set
> > &gt; &amp;gt; &amp;amp;gt; to
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy
> type, the task are
> > executed as a dummy
> > &gt; task and the
> > &gt; &amp;gt; real task
> > &gt; &amp;gt; &amp;amp;gt; is not
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; executed.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;amp;amp;gt; By the way，I
> > &gt; &amp;gt; &amp;amp;gt; already realize these
> > two&amp;amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; features
> in my fork
> > &gt; branch.&amp;amp;amp;amp;amp;nbsp;Whether the
> > &gt; &amp;gt; follow-up
> > &gt; &amp;gt; &amp;amp;gt; release can be
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; supported
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> >
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> <SSHClient.java&amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;
> > &gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;gt;
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
Good idea! It's can solve the exceptions where the numbers of SSH connections exceeds.
We also have another problem of how to config remote server easily when we have hundreds of tasks.
Now I use the local param to config remote server's IP,username,password. But every task needs to config.
------------------ 原始邮件 ------------------
发件人: "lidong dai"<dailidong66@gmail.com>;
发送时间: 2020年5月26日(星期二) 晚上6:38
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
good idea, I think "Control the number of task's concurrent" may be have
some other methods, for now, we don't have this feature, but when I
communicate with the team of China Unicom， they provide a new solve
solution：
we can add a task concurrent page, you can set a variable value like
'ETL_JOB' =10 and so on, then give the task type add a option for choosing
ETL_JOB or other variable, and there may be many task also choose ETL_JOB
variable. when these tasks begin to running at the same time, the Master
Server will check whether a task begin to run or wait(if the running tasks
num = 10)
Best egards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <peilongwu@qq.com> 于2020年5月26日周二 下午2:34写道：
> There is only two ways that I can think to do about this.
>
>
> 1. Simple SSH connect pool
> 2. Control the number of task's concurrent
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> 发送时间:&nbsp;2020年5月22日(星期五) 晚上7:29
> 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
>
> 主题:&nbsp;Re: [Feature] Support SSH Task
>
>
>
> What other friends think about this discuss
>
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen&nbsp; 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <peilongwu@qq.com&gt; 于2020年5月22日周五 上午10:20写道：
>
> &gt; Your are right, it can be solved the problem by modifying the
> &gt; number of server ssh connections.&amp;nbsp;
> &gt;
> &gt; The real reason is that I want to control the&amp;nbsp; number of SSH
> &gt; connections and reuse the SSH connection, Because the remote server is
> &gt; managed by other team.
> &gt;
> &gt; Otherwise, I think that we can also control the number of parallel
> tasks
> &gt; in a workflow.
> &gt;
> &gt;
> &gt; It is a good idea that we can use the encapsulated ssh connection to
> get
> &gt; clear
> &gt; exception information&amp;nbsp;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:&amp;nbsp;"Hemin Wen"<wenhemin@apache.org&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2020年5月21日(星期四) 中午11:07
> &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org&amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> &gt;
> &gt;
> &gt;
> &gt; Thanks.
> &gt;
> &gt; About ssh connections pool,
> &gt; 1. If the connection is not enough, Can it be solved by modifying the
> &gt; number of server ssh connections.
> &gt; 2. I think we have to look at whether a server's parallel ssh tasks
> will
> &gt; reach the bottleneck of server connections.
> &gt;
> &gt; I think it is better to use the encapsulated ssh connection to get
> clear
> &gt; exception information.
> &gt;
> &gt; --------------------
> &gt; DolphinScheduler(Incubator) Commtter
> &gt; Hemin Wen&amp;nbsp; 温合民
> &gt; wenhemin@apache.org
> &gt; --------------------
> &gt;
> &gt;
> &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午11:29写道：
> &gt;
> &gt; &amp;gt; Thanks for your advise and suggestions. It may has some
> exceptions if
> &gt; I
> &gt; &amp;gt; implement SSH Task as you said. Now, let me explain my ideas.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; e.g
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; My workflow DAG has 100 or more task, it's all ssh task. I
> &gt; have&amp;amp;nbsp; two
> &gt; &amp;gt; questions.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; 1、We must limit SSH connection and reuse the SSH connections,
> &gt; otherwise
> &gt; &amp;gt; the remote server will be refused.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; 2、If we config all of these task, we have a lot of repeat
> work to do.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; By the way, the way I'm currently verifying is I use local
> params to
> &gt; &amp;gt; config the remote server's host,user,password
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<dailidong66@gmail.com
> &amp;amp;gt;;
> &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56
> &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;amp;gt;;
> &gt; &amp;gt;
> &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH Task
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; hi, I see you will Jsch, does his license compitable with
> apache v2
> &gt; &amp;gt; license, I see the following words "
> &gt; &amp;gt; JSch 0.0.* was released under the GNU LGPL license. Later,
> we have
> &gt; switched
> &gt; &amp;gt; over to a BSD-style license.
> &gt; &amp;gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt .
> you know,
> &gt; LGPL
> &gt; &amp;gt; license not compitable with apache license. but BSD is ok ,
> so the
> &gt; version
> &gt; &amp;gt; of Jsch is key point
> &gt; &amp;gt;
> &gt; &amp;gt; second, Connection Pool is heavy resource, Please use
> carefully
> &gt; &amp;gt;
> &gt; &amp;gt; and I'm a little confused, why there must be add a new
> &gt; Table,&amp;amp;nbsp; other
> &gt; &amp;gt; task
> &gt; &amp;gt; types also no seperate table,&amp;amp;nbsp; could you
> consider task_type
> &gt; and
> &gt; &amp;gt; task_json
> &gt; &amp;gt; in t_ds_task_instance ?
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Best Regards
> &gt; &amp;gt; ---------------
> &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> &gt; &amp;gt; Lidong Dai 代立冬
> &gt; &amp;gt; dailidong66@gmail.com
> &gt; &amp;gt; ---------------
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt; 于2020年5月20日周三 下午9:57写道：
> &gt; &amp;gt;
> &gt; &amp;gt; &amp;amp;gt; I want to describe my idea.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; First, we must design a host table that's named
> &gt; t_ds_ssh_host, e.g
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; - id
> &gt; &amp;gt; &amp;amp;gt; - name
> &gt; &amp;gt; &amp;amp;gt; - ip / host
> &gt; &amp;gt; &amp;amp;gt; - user
> &gt; &amp;gt; &amp;amp;gt; - password
> &gt; &amp;gt; &amp;amp;gt; - max_connection
> &gt; &amp;gt; &amp;amp;gt; - create_time
> &gt; &amp;gt; &amp;amp;gt; - update_time
> &gt; &amp;gt; &amp;amp;gt; - user_id
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Second, Shell can execute in worker server or
> remote host.
> &gt; We can
> &gt; &amp;gt; choose a
> &gt; &amp;gt; &amp;amp;gt; host in host's list.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Third, when workflow running, we must maintain
> the remote
> &gt; host's
> &gt; &amp;gt; &amp;amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Finally, when workflow finish, we must release
> the
> &gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Why do we have to maintain a connection pool.
> Because it is
> &gt; very easy
> &gt; &amp;gt; to
> &gt; &amp;gt; &amp;amp;gt; get exceptions when it's has many SSH task.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> &gt; &amp;gt; &amp;amp;gt; 发件人:&amp;amp;amp;nbsp;"lidong dai"<
> dailidong66@gmail.com
> &gt; &amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> &gt; &amp;gt; &amp;amp;gt; 收件人:&amp;amp;amp;nbsp;"dev"<
> dev@dolphinscheduler.apache.org
> &gt; &amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re: [Feature] Support SSH
> Task
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; glad to hear that you will implement this
> feature
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Best Regards
> &gt; &amp;gt; &amp;amp;gt; ---------------
> &gt; &amp;gt; &amp;amp;gt; DolphinScheduler(Incubator) PPMC
> &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬
> &gt; &amp;gt; &amp;amp;gt; dailidong66@gmail.com
> &gt; &amp;gt; &amp;amp;gt; ---------------
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;amp;gt;
> 于2020年5月20日周三 下午3:47写道：
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; My code is not perfect yet. I
> will write a
> &gt; detailed design
> &gt; &amp;gt; document.
> &gt; &amp;gt; &amp;amp;gt; Then
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I will realize this feature
> about our
> &gt; discussion result.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 发件人:&amp;amp;amp;amp;nbsp;"wenhemin"<
> &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 发送时间:&amp;amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 收件人:&amp;amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> &gt; &amp;amp;amp;amp;gt;;"dev"<
> &gt; &amp;gt; &amp;amp;gt; dev@dolphinscheduler.apache.org
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题:&amp;amp;amp;amp;nbsp;Re:
> [Feature] Support SSH
> &gt; Task and Support
> &gt; &amp;gt; dummy task
> &gt; &amp;gt; &amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Thanks for writing detailed
> documentation. I
> &gt; think this is
> &gt; &amp;gt; also a
> &gt; &amp;gt; &amp;amp;gt; missing
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; feature of DS.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About the extension point:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 1.Can ssh tasks be merged into
> shell tasks.
> &gt; Essentially,
> &gt; &amp;gt; they all
> &gt; &amp;gt; &amp;amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; shell commands.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 2.About dummy task, DS has the
> function of
> &gt; disable nodes, I
> &gt; &amp;gt; do n’t
> &gt; &amp;gt; &amp;amp;gt; know if
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this requirement is met.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; The script from AirFlow to
> Dolphin is great.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 在
> 2020年5月18日，09:28，裴龙武 <
> &gt; peilongwu@qq.com&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; OK, 3Q!
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; First, I
> will ensure that open
> &gt; source can use.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; Second, I
> think we must
> &gt; discuss deeply. I write
> &gt; &amp;gt; a more
> &gt; &amp;gt; &amp;amp;gt; detailed
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; document. You can check the
> attachment. I also
> &gt; send the
> &gt; &amp;gt; document to
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DaiLidong.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> Third,&amp;amp;amp;amp;nbsp; I'll
> &gt; give you the error of
> &gt; &amp;gt; not using SSH
> &gt; &amp;gt; &amp;amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> ------------------ 原始邮件
> &gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 发件人:
> "wenhemin"<
> &gt; whm_777@163.com&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 发送时间:
> 2020年5月14日(星期四) 晚上7:26
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 收件人:
> "裴龙武"<peilongwu@qq.com
> &gt; &amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; 主题: Re:
> [Feature] Support SSH
> &gt; Task and Support
> &gt; &amp;gt; dummy task
> &gt; &amp;gt; &amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; Great!
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; I think,
> Can ssh tasks be
> &gt; merged into shell
> &gt; &amp;gt; tasks,&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script locally or remotely,
> Configure on the
> &gt; front end.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; About ssh
> connect pool, I did
> &gt; not find it
> &gt; &amp;gt; necessary to use
> &gt; &amp;gt; &amp;amp;gt; the
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt; BTW, Look
> at the code to
> &gt; introduce additional
> &gt; &amp;gt; jar packages,
> &gt; &amp;gt; &amp;amp;gt; You also
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; need to ensure that open
> source can use the
> &gt; license of this
> &gt; &amp;gt; jar
> &gt; &amp;gt; &amp;amp;gt; package.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在
> &gt; 2020年5月14日，16:20，裴龙武 <
> &gt; &amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1. The
> &gt; priority between these tasks
> &gt; &amp;gt; is also depended
> &gt; &amp;gt; &amp;amp;gt; on the
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dolphin DAG define. When the
> front task is not
> &gt; finished, it
> &gt; &amp;gt; not
> &gt; &amp;gt; &amp;amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; next task.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2. I extend
> &gt; ssh task. I also use
> &gt; &amp;gt; local params to
> &gt; &amp;gt; &amp;amp;gt; config ssh host,
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user and password.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; E.g:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; public static
> &gt; AbstractTask
> &gt; &amp;gt; &amp;amp;gt; newTask(TaskExecutionContext
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; taskExecutionContext, Logger
> logger)
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; throws
> &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException {
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> Boolean
> &gt; &amp;gt; enable =
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> if
> &gt; &amp;gt; (enable != null
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;
> enable == false ) {
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> DummyTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> }
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> switch
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case SHELL:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ShellTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case
> &gt; &amp;gt; &amp;amp;gt; PROCEDURE:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ProcedureTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case SQL:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SqlTask(taskExecutionContext,
> logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case MR:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> MapReduceTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case SPARK:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> SparkTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case FLINK:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> FlinkTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case PYTHON:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> PythonTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case HTTP:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; HttpTask(taskExecutionContext,
> logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case DATAX:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> DataxTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case SQOOP:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> SqoopTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; case SSH:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSHTask(taskExecutionContext,
> logger);
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; default:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; logger.error("unsupport task
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; type: {}",
> taskExecutionContext.getTaskType());
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; throw new
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; IllegalArgumentException("not
> support task
> &gt; type");
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> }
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; }
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3. I am not
> &gt; sure that it supports
> &gt; &amp;gt; window or not.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; ------------------ 原始邮件
> &gt; &amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发件人:
> &gt; "wenhemin"<whm_777@163.com
> &gt; &amp;gt; <mailto:
> &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发送时间:
> &gt; 2020年5月14日(星期四) 下午3:46
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 收件人: "裴龙武"<
> &gt; peilongwu@qq.com
> &gt; &amp;gt; <mailto:peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 主题: Re:
> &gt; [Feature] Support SSH Task
> &gt; &amp;gt; and Support dummy
> &gt; &amp;gt; &amp;amp;gt; task like
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; Sorry, My
> &gt; previous description is
> &gt; &amp;gt; not very clear.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I want to ask
> &gt; some questions:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1.How to
> &gt; control the priority
> &gt; &amp;gt; between ssh tasks?
> &gt; &amp;gt; &amp;amp;gt; There may be
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; some ssh tasks that have been
> waiting for
> &gt; execution.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2.I understand
> &gt; what you want to
> &gt; &amp;gt; solve is the problem
> &gt; &amp;gt; &amp;amp;gt; of executing
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; remote ssh scripts in batches.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> So, not
> &gt; &amp;gt; sure how to use this
> &gt; &amp;gt; &amp;amp;gt; function.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3.I don't know
> &gt; if this supports
> &gt; &amp;gt; windows system.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在
> 2020年5月13日，20:56，裴龙武 <
> &gt; &amp;gt; peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; <mailto:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I use
> spin lock. Here
> &gt; &amp;gt; is my code. Of course
> &gt; &amp;gt; &amp;amp;gt; , it's not
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perfect. I just do a test. To
> my surprise, it
> &gt; is the result
> &gt; &amp;gt; of the
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execution is the same as the
> AirFlow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 相同的结果。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> ------------------ 原始邮件
> &gt; &amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发件人:
> "whm_777"<
> &gt; &amp;gt; whm_777@163.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发送时间:
> 2020年5月13日(星期三)
> &gt; &amp;gt; 晚上7:21
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 收件人:
> "裴龙武"<
> &gt; &amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 主题: Re:
> [Feature]
> &gt; &amp;gt; Support SSH Task and
> &gt; &amp;gt; &amp;amp;gt; Support dummy task
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; You can
> modify the
> &gt; &amp;gt; maximum number of linux
> &gt; &amp;gt; &amp;amp;gt; ssh connections.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; If use
> ssh connection
> &gt; &amp;gt; pool, How to control
> &gt; &amp;gt; &amp;amp;gt; the priority of
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh?
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 在
> &gt; &amp;gt; 2020年5月13日，18:01，裴龙武 <
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> First 3Q，
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; I&amp;amp;amp;amp;nbsp; use more than 100 task
> &gt; &amp;gt; &amp;amp;gt; node. But SSH connections
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are limited.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; <330BE502@F7F80E73.76C5BB5E.jpg
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; ------------------ 原始邮件
> &gt; &amp;gt; &amp;amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 发件人:
> &gt; &amp;gt; "whm_777"<whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt; <mailto:whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 发送时间:
> &gt; &amp;gt; 2020年5月13日(星期三) 下午5:50
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 收件人: "裴龙武"<
> &gt; &amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> 主题: Re:
> &gt; &amp;gt; [Feature] Support SSH Task
> &gt; &amp;gt; &amp;amp;gt; and Support dummy task
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> E.g.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; rtn_code=`ssh -o
> &gt; &amp;gt; &amp;amp;gt; ServerAliveInterval=60 -p xxxx
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; root@xxx.xxx.xxx.xxx <mailto:
> &gt; root@xxx.xxx.xxx.xxx&amp;amp;amp;amp;gt;
> &gt; &amp;gt; ‘shell
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; command&amp;amp;amp;amp;nbsp;
> &gt; &amp;amp;amp;amp;gt;/dev/null
> &gt; &amp;gt; 2&amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;1; echo $?'`
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> if [
> &gt; &amp;gt; "$rtn_code" -eq 0 ]; then
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; echo "成功"
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; exit 0
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> else
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; echo "失败"
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; exit 1
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> fi
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> Batch shell
> &gt; &amp;gt; command is not supported.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> Multiple
> &gt; &amp;gt; servers can be split into
> &gt; &amp;gt; &amp;amp;gt; multiple task nodes.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; Could you give me a
> &gt; &amp;gt; &amp;amp;gt; example，3Q. 能否给我一个例子，谢谢！
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; By the way, I have more than
> &gt; &amp;gt; &amp;amp;gt; 100 tasks in one DAG.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; These tasks connect two other
> server to
> &gt; execute. So SSH
> &gt; &amp;gt; tasks must
> &gt; &amp;gt; &amp;amp;gt; have
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; pool to manager. Now I use
> JSch and realize a
> &gt; simple pool.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> &gt; &amp;gt; &amp;amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用
> JSch，并实现了一个简单的连接池。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; ------------------ 原始邮件
> &gt; &amp;gt; &amp;amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 发件人: "wenhemin"<
> &gt; &amp;gt; &amp;amp;gt; whm_777@163.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; whm_777@163.com
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 收件人: "dev"<
> &gt; &amp;gt; &amp;amp;gt; dev@dolphinscheduler.apache.org <mailto:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> dev@dolphinscheduler.apache.org
> &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; 主题: Re: [Feature] Support
> &gt; &amp;gt; &amp;amp;gt; SSH Task and Support dummy
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; The shell node is supports
> &gt; &amp;gt; &amp;amp;gt; remote calling, and get
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the remote command result code.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; 在
> &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto:peilongwu@qq.com
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Dear ALL：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Support Linux SSH
> &gt; &amp;gt; &amp;amp;gt; Task 支持 Linux SSH 任务
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker
> &gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> &gt; &amp;gt; &amp;amp;gt; project, the workflow's tasks
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; want to execute shell scripts
> where are in
> &gt; different server's
> &gt; &amp;gt; &amp;amp;gt; different
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; directory. When worker execute
> these shell
> &gt; scripts, it must
> &gt; &amp;gt; use the
> &gt; &amp;gt; &amp;amp;gt; same
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user to login these server.
> Also, the worker
> &gt; can get the
> &gt; &amp;gt; executing
> &gt; &amp;gt; &amp;amp;gt; state of
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; these server. We can config
> these server 's
> &gt; host,user and
> &gt; &amp;gt; password.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; SSH Task is very
> &gt; &amp;gt; &amp;amp;gt; useful for most user SSH
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 任务对大多数用户是非常有用的
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> &gt; Worker
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; In
> &gt; &amp;gt; &amp;amp;gt; dolphinscheduler, the most executing tasks
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are in different servers who
> are not workers.
> &gt; These servers
> &gt; &amp;gt; also have
> &gt; &amp;gt; &amp;amp;gt; their
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; different fixed services. We
> just have to pass
> &gt; different
> &gt; &amp;gt; parameters to
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; schedule these shell scripts
> to execute.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> &gt; &amp;gt; &amp;amp;gt; to execute ssh script Python
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Python
> &gt; &amp;gt; &amp;amp;gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module
> &gt; &amp;gt; &amp;amp;gt; that can execute SSH Shell
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script. It's paramiko.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Others 其他内容
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; I found this
> &gt; &amp;gt; &amp;amp;gt; described in previous feature, but
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; it was relatively simple.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Feature URL
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; 另外，我通过 Shell Task
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; In addition, it is
> &gt; &amp;gt; &amp;amp;gt; very inconvenient for me to
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perform remote tasks through
> Shell Task. Here
> &gt; is my script.
> &gt; &amp;gt; I don't
> &gt; &amp;gt; &amp;amp;gt; know if
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; there's a better way.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; sshpass -p
> &gt; &amp;gt; &amp;amp;gt; 'password' ssh user@host echo 'ssh
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; success' echo 'Hello World'
> &gt; -&amp;amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; /home/dolphinscheduler/test/hello.txt
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo 'end'
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Support dummy task
> &gt; &amp;gt; &amp;amp;gt; like airflow 支持像 Airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 中的虚拟任务
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> &gt; &amp;gt; &amp;amp;gt; DAG 文件，DAG
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt;
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> &gt; &amp;gt; &amp;amp;gt; Dummy
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my
> &gt; &amp;gt; &amp;amp;gt; project, it has a productized
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG file. The file contains
> different modules,
> &gt; some of which
> &gt; &amp;gt; are
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; interdependent and some of
> which are not. When
> &gt; customers
> &gt; &amp;gt; purchase
> &gt; &amp;gt; &amp;amp;gt; different
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; modules, we need to set some
> tasks as dummy
> &gt; tasks, which
> &gt; &amp;gt; some modules
> &gt; &amp;gt; &amp;amp;gt; are
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; not purchased and the
> purchased module is not
> &gt; dependent.
> &gt; &amp;gt; Because of
> &gt; &amp;gt; &amp;amp;gt; this
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setting, these dummy tasks are
> actually not
> &gt; executed. The
> &gt; &amp;gt; benefits of
> &gt; &amp;gt; &amp;amp;gt; this
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setup are product unity and
> diagram integrity.
> &gt; In airflow,
> &gt; &amp;gt; these task
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute by dummy operator.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; ** Realize 实现方式**
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task
> &gt; &amp;gt; &amp;amp;gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task is easy
> &gt; &amp;gt; &amp;amp;gt; to realize, but it need to
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; use with other different
> tasks. When the task's
> &gt; executed
> &gt; &amp;gt; type is set
> &gt; &amp;gt; &amp;amp;gt; to
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy type, the task are
> executed as a dummy
> &gt; task and the
> &gt; &amp;gt; real task
> &gt; &amp;gt; &amp;amp;gt; is not
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; executed.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;amp;amp;gt; By the way，I
> &gt; &amp;gt; &amp;amp;gt; already realize these
> two&amp;amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; features in my fork
> &gt; branch.&amp;amp;amp;amp;amp;nbsp;Whether the
> &gt; &amp;gt; follow-up
> &gt; &amp;gt; &amp;amp;gt; release can be
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; supported
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> <SSHClient.java&amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; &amp;amp;amp;amp;gt;
> &gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;gt;
Re: [Feature] Support SSH Task
Posted by Hemin Wen <we...@apache.org>.
Good idea!
This can control the number of concurrency for each task type.
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
lidong dai <da...@gmail.com> 于2020年5月26日周二 下午6:39写道：
> good idea, I think "Control the number of task's concurrent" may be have
> some other methods, for now, we don't have this feature, but when I
> communicate with the team of China Unicom， they provide a new solve
> solution：
> we can add a task concurrent page, you can set a variable value like
> 'ETL_JOB' =10 and so on, then give the task type add a option for choosing
> ETL_JOB or other variable, and there may be many task also choose ETL_JOB
> variable. when these tasks begin to running at the same time, the Master
> Server will check whether a task begin to run or wait(if the running tasks
> num = 10)
>
>
> Best egards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <pe...@qq.com> 于2020年5月26日周二 下午2:34写道：
>
> > There is only two ways that I can think to do about this.
> >
> >
> > 1. Simple SSH connect pool
> > 2. Control the number of task's concurrent
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "Hemin Wen"<wenhemin@apache.org>;
> > 发送时间: 2020年5月22日(星期五) 晚上7:29
> > 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> >
> > 主题: Re: [Feature] Support SSH Task
> >
> >
> >
> > What other friends think about this discuss
> >
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > 裴龙武 <peilongwu@qq.com> 于2020年5月22日周五 上午10:20写道：
> >
> > > Your are right, it can be solved the problem by modifying the
> > > number of server ssh connections.&nbsp;
> > >
> > > The real reason is that I want to control the&nbsp; number of
> SSH
> > > connections and reuse the SSH connection, Because the remote server
> is
> > > managed by other team.
> > >
> > > Otherwise, I think that we can also control the number of parallel
> > tasks
> > > in a workflow.
> > >
> > >
> > > It is a good idea that we can use the encapsulated ssh connection to
> > get
> > > clear
> > > exception information&nbsp;
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> > > 发送时间:&nbsp;2020年5月21日(星期四) 中午11:07
> > > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> > >
> > > 主题:&nbsp;Re: [Feature] Support SSH Task
> > >
> > >
> > >
> > > Thanks.
> > >
> > > About ssh connections pool,
> > > 1. If the connection is not enough, Can it be solved by modifying
> the
> > > number of server ssh connections.
> > > 2. I think we have to look at whether a server's parallel ssh tasks
> > will
> > > reach the bottleneck of server connections.
> > >
> > > I think it is better to use the encapsulated ssh connection to get
> > clear
> > > exception information.
> > >
> > > --------------------
> > > DolphinScheduler(Incubator) Commtter
> > > Hemin Wen&nbsp; 温合民
> > > wenhemin@apache.org
> > > --------------------
> > >
> > >
> > > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午11:29写道：
> > >
> > > &gt; Thanks for your advise and suggestions. It may has some
> > exceptions if
> > > I
> > > &gt; implement SSH Task as you said. Now, let me explain my
> ideas.
> > > &gt;
> > > &gt;
> > > &gt; e.g
> > > &gt;
> > > &gt;
> > > &gt; My workflow DAG has 100 or more task, it's all ssh task. I
> > > have&amp;nbsp; two
> > > &gt; questions.
> > > &gt;
> > > &gt;
> > > &gt; 1、We must limit SSH connection and reuse the SSH
> connections,
> > > otherwise
> > > &gt; the remote server will be refused.
> > > &gt;
> > > &gt;
> > > &gt; 2、If we config all of these task, we have a lot of repeat
> > work to do.
> > > &gt;
> > > &gt;
> > > &gt; By the way, the way I'm currently verifying is I use local
> > params to
> > > &gt; config the remote server's host,user,password
> > > &gt;
> > > &gt;
> > > &gt;
> > ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > > &gt; 发件人:&amp;nbsp;"lidong dai"<dailidong66@gmail.com
> > &amp;gt;;
> > > &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> > &amp;gt;;
> > > &gt;
> > > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt; hi, I see you will Jsch, does his license compitable with
> > apache v2
> > > &gt; license, I see the following words "
> > > &gt; JSch 0.0.* was released under the GNU LGPL license. Later,
> > we have
> > > switched
> > > &gt; over to a BSD-style license.
> > > &gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt .
> > you know,
> > > LGPL
> > > &gt; license not compitable with apache license. but BSD is ok ,
> > so the
> > > version
> > > &gt; of Jsch is key point
> > > &gt;
> > > &gt; second, Connection Pool is heavy resource, Please use
> > carefully
> > > &gt;
> > > &gt; and I'm a little confused, why there must be add a new
> > > Table,&amp;nbsp; other
> > > &gt; task
> > > &gt; types also no seperate table,&amp;nbsp; could you
> > consider task_type
> > > and
> > > &gt; task_json
> > > &gt; in t_ds_task_instance ?
> > > &gt;
> > > &gt;
> > > &gt;
> > > &gt; Best Regards
> > > &gt; ---------------
> > > &gt; DolphinScheduler(Incubator) PPMC
> > > &gt; Lidong Dai 代立冬
> > > &gt; dailidong66@gmail.com
> > > &gt; ---------------
> > > &gt;
> > > &gt;
> > > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午9:57写道：
> > > &gt;
> > > &gt; &amp;gt; I want to describe my idea.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; First, we must design a host table that's
> named
> > > t_ds_ssh_host, e.g
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; - id
> > > &gt; &amp;gt; - name
> > > &gt; &amp;gt; - ip / host
> > > &gt; &amp;gt; - user
> > > &gt; &amp;gt; - password
> > > &gt; &amp;gt; - max_connection
> > > &gt; &amp;gt; - create_time
> > > &gt; &amp;gt; - update_time
> > > &gt; &amp;gt; - user_id
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Second, Shell can execute in worker server or
> > remote host.
> > > We can
> > > &gt; choose a
> > > &gt; &amp;gt; host in host's list.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Third, when workflow running, we must maintain
> > the remote
> > > host's
> > > &gt; &amp;gt; connection pool.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Finally, when workflow finish, we must release
> > the
> > > connection pool.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Why do we have to maintain a connection pool.
> > Because it is
> > > very easy
> > > &gt; to
> > > &gt; &amp;gt; get exceptions when it's has many SSH task.
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > >
> >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<
> > dailidong66@gmail.com
> > > &amp;amp;gt;;
> > > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<
> > dev@dolphinscheduler.apache.org
> > > &amp;amp;gt;;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH
> > Task
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; glad to hear that you will implement this
> > feature
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; Best Regards
> > > &gt; &amp;gt; ---------------
> > > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> > > &gt; &amp;gt; Lidong Dai 代立冬
> > > &gt; &amp;gt; dailidong66@gmail.com
> > > &gt; &amp;gt; ---------------
> > > &gt; &amp;gt;
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt;
> > 于2020年5月20日周三 下午3:47写道：
> > > &gt; &amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; My code is not perfect yet. I
> > will write a
> > > detailed design
> > > &gt; document.
> > > &gt; &amp;gt; Then
> > > &gt; &amp;gt; &amp;amp;gt; I will realize this feature
> > about our
> > > discussion result.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > 发件人:&amp;amp;amp;nbsp;"wenhemin"<
> > > whm_777@163.com&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > 发送时间:&amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > > &gt; &amp;gt; &amp;amp;gt;
> > 收件人:&amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> > > &amp;amp;amp;gt;;"dev"<
> > > &gt; &amp;gt; dev@dolphinscheduler.apache.org
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re:
> > [Feature] Support SSH
> > > Task and Support
> > > &gt; dummy task
> > > &gt; &amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; Thanks for writing detailed
> > documentation. I
> > > think this is
> > > &gt; also a
> > > &gt; &amp;gt; missing
> > > &gt; &amp;gt; &amp;amp;gt; feature of DS.
> > > &gt; &amp;gt; &amp;amp;gt; About the extension point:
> > > &gt; &amp;gt; &amp;amp;gt; 1.Can ssh tasks be merged
> into
> > shell tasks.
> > > Essentially,
> > > &gt; they all
> > > &gt; &amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; shell commands.
> > > &gt; &amp;gt; &amp;amp;gt; 2.About dummy task, DS has
> the
> > function of
> > > disable nodes, I
> > > &gt; do n’t
> > > &gt; &amp;gt; know if
> > > &gt; &amp;gt; &amp;amp;gt; this requirement is met.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; The script from AirFlow to
> > Dolphin is great.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 在
> > 2020年5月18日，09:28，裴龙武 <
> > > peilongwu@qq.com&amp;amp;amp;gt;
> > > &gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; OK, 3Q!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, I
> > will ensure that open
> > > source can use.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second,
> I
> > think we must
> > > discuss deeply. I write
> > > &gt; a more
> > > &gt; &amp;gt; detailed
> > > &gt; &amp;gt; &amp;amp;gt; document. You can check the
> > attachment. I also
> > > send the
> > > &gt; document to
> > > &gt; &amp;gt; &amp;amp;gt; DaiLidong.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > Third,&amp;amp;amp;nbsp; I'll
> > > give you the error of
> > > &gt; not using SSH
> > > &gt; &amp;gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > ------------------ 原始邮件
> > > ------------------
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发件人:
> > "wenhemin"<
> > > whm_777@163.com&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发送时间:
> > 2020年5月14日(星期四) 晚上7:26
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 收件人:
> > "裴龙武"<peilongwu@qq.com
> > > &amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题: Re:
> > [Feature] Support SSH
> > > Task and Support
> > > &gt; dummy task
> > > &gt; &amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Great!
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I think,
> > Can ssh tasks be
> > > merged into shell
> > > &gt; tasks,&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; script locally or remotely,
> > Configure on the
> > > front end.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About
> ssh
> > connect pool, I did
> > > not find it
> > > &gt; necessary to use
> > > &gt; &amp;gt; the
> > > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; BTW,
> Look
> > at the code to
> > > introduce additional
> > > &gt; jar packages,
> > > &gt; &amp;gt; You also
> > > &gt; &amp;gt; &amp;amp;gt; need to ensure that open
> > source can use the
> > > license of this
> > > &gt; jar
> > > &gt; &amp;gt; package.
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 在
> > > 2020年5月14日，16:20，裴龙武 <
> > > &gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; 写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 1. The
> > > priority between these tasks
> > > &gt; is also depended
> > > &gt; &amp;gt; on the
> > > &gt; &amp;gt; &amp;amp;gt; dolphin DAG define. When the
> > front task is not
> > > finished, it
> > > &gt; not
> > > &gt; &amp;gt; execute
> > > &gt; &amp;gt; &amp;amp;gt; next task.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 2. I extend
> > > ssh task. I also use
> > > &gt; local params to
> > > &gt; &amp;gt; config ssh host,
> > > &gt; &amp;gt; &amp;amp;gt; user and password.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; E.g:
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; public static
> > > AbstractTask
> > > &gt; &amp;gt; newTask(TaskExecutionContext
> > > &gt; &amp;gt; &amp;amp;gt; taskExecutionContext, Logger
> > logger)
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; throws
> > > &gt; &amp;gt; IllegalArgumentException {
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > Boolean
> > > &gt; enable =
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > if
> > > &gt; (enable != null
> > > &gt; &amp;gt; &amp;amp;amp;amp;&amp;amp;amp;amp;
> > enable == false ) {
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > DummyTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > }
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > switch
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case SHELL:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > ShellTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case
> > > &gt; &amp;gt; PROCEDURE:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > ProcedureTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case SQL:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; SqlTask(taskExecutionContext,
> > logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case MR:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > MapReduceTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case SPARK:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > SparkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case FLINK:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > FlinkTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case PYTHON:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > PythonTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case HTTP:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> HttpTask(taskExecutionContext,
> > logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case DATAX:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > DataxTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case SQOOP:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt;
> > SqoopTask(taskExecutionContext, logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; case SSH:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; return new
> > > &gt; &amp;gt; &amp;amp;gt; SSHTask(taskExecutionContext,
> > logger);
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; default:
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; logger.error("unsupport task
> > > &gt; &amp;gt; &amp;amp;gt; type: {}",
> > taskExecutionContext.getTaskType());
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; throw new
> > > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException("not
> > support task
> > > type");
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > }
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; }
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 3. I am not
> > > sure that it supports
> > > &gt; window or not.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > ------------------ 原始邮件
> > > &gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> > > "wenhemin"<whm_777@163.com
> > > &gt; <mailto:
> > > &gt; &amp;gt; whm_777@163.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> > > 2020年5月14日(星期四) 下午3:46
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"<
> > > peilongwu@qq.com
> > > &gt; <mailto:peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> > > [Feature] Support SSH Task
> > > &gt; and Support dummy
> > > &gt; &amp;gt; task like
> > > &gt; &amp;gt; &amp;amp;gt; airflow
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; Sorry, My
> > > previous description is
> > > &gt; not very clear.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; I want to ask
> > > some questions:
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 1.How to
> > > control the priority
> > > &gt; between ssh tasks?
> > > &gt; &amp;gt; There may be
> > > &gt; &amp;gt; &amp;amp;gt; some ssh tasks that have been
> > waiting for
> > > execution.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 2.I understand
> > > what you want to
> > > &gt; solve is the problem
> > > &gt; &amp;gt; of executing
> > > &gt; &amp;gt; &amp;amp;gt; remote ssh scripts in
> batches.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > So, not
> > > &gt; sure how to use this
> > > &gt; &amp;gt; function.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt; 3.I don't know
> > > if this supports
> > > &gt; windows system.
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 在
> > 2020年5月13日，20:56，裴龙武 <
> > > &gt; peilongwu@qq.com
> > > &gt; &amp;gt; <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > 写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; I use
> > spin lock. Here
> > > &gt; is my code. Of course
> > > &gt; &amp;gt; , it's not
> > > &gt; &amp;gt; &amp;amp;gt; perfect. I just do a test. To
> > my surprise, it
> > > is the result
> > > &gt; of the
> > > &gt; &amp;gt; &amp;amp;gt; execution is the same as the
> > AirFlow
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > > &gt; &amp;gt; &amp;amp;gt; 相同的结果。
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > ------------------ 原始邮件
> > > &gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> > "whm_777"<
> > > &gt; whm_777@163.com <mailto:
> > > &gt; &amp;gt; whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> > 2020年5月13日(星期三)
> > > &gt; 晚上7:21
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 收件人:
> > "裴龙武"<
> > > &gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> > [Feature]
> > > &gt; Support SSH Task and
> > > &gt; &amp;gt; Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; You can
> > modify the
> > > &gt; maximum number of linux
> > > &gt; &amp;gt; ssh connections.
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; If use
> > ssh connection
> > > &gt; pool, How to control
> > > &gt; &amp;gt; the priority of
> > > &gt; &amp;gt; &amp;amp;gt; ssh?
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > 在
> > > &gt; 2020年5月13日，18:01，裴龙武 <
> > > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > 写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > First 3Q，
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; I&amp;amp;amp;nbsp; use more than 100 task
> > > &gt; &amp;gt; node. But SSH connections
> > > &gt; &amp;gt; &amp;amp;gt; are limited.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; <330BE502@F7F80E73.76C5BB5E.jpg
> > > &gt; &amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; ------------------ 原始邮件
> > > &gt; &amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > 发件人:
> > > &gt; "whm_777"<whm_777@163.com
> > > &gt; &amp;gt; <mailto:whm_777@163.com
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > 发送时间:
> > > &gt; 2020年5月13日(星期三) 下午5:50
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > 收件人: "裴龙武"<
> > > &gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > 主题: Re:
> > > &gt; [Feature] Support SSH Task
> > > &gt; &amp;gt; and Support dummy task
> > > &gt; &amp;gt; &amp;amp;gt; like airflow
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > E.g.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; rtn_code=`ssh -o
> > > &gt; &amp;gt; ServerAliveInterval=60 -p xxxx
> > > &gt; &amp;gt; &amp;amp;gt; root@xxx.xxx.xxx.xxx
> <mailto:
> > > root@xxx.xxx.xxx.xxx&amp;amp;amp;gt;
> > > &gt; ‘shell
> > > &gt; &amp;gt; &amp;amp;gt; command&amp;amp;amp;nbsp;
> > > &amp;amp;amp;gt;/dev/null
> > > &gt; 2&amp;amp;amp;gt;&amp;amp;amp;amp;1; echo $?'`
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > if [
> > > &gt; "$rtn_code" -eq 0 ]; then
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; echo "成功"
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; exit 0
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > else
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; echo "失败"
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > > &gt; &amp;gt; exit 1
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > fi
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > Batch shell
> > > &gt; command is not supported.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > Multiple
> > > &gt; servers can be split into
> > > &gt; &amp;gt; multiple task nodes.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 在 2020年5月13日，17:40，裴龙武 <
> > > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > 写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; Could you give me a
> > > &gt; &amp;gt; example，3Q. 能否给我一个例子，谢谢！
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; By the way, I have more than
> > > &gt; &amp;gt; 100 tasks in one DAG.
> > > &gt; &amp;gt; &amp;amp;gt; These tasks connect two other
> > server to
> > > execute. So SSH
> > > &gt; tasks must
> > > &gt; &amp;gt; have
> > > &gt; &amp;gt; &amp;amp;gt; pool to manager. Now I use
> > JSch and realize a
> > > simple pool.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > > &gt; &amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > > &gt; &amp;gt; &amp;amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用
> > JSch，并实现了一个简单的连接池。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; ------------------ 原始邮件
> > > &gt; &amp;gt; ------------------
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 发件人: "wenhemin"<
> > > &gt; &amp;gt; whm_777@163.com <mailto:
> > > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 收件人: "dev"<
> > > &gt; &amp;gt; dev@dolphinscheduler.apache.org <mailto:
> > > &gt; &amp;gt; &amp;amp;gt;
> > dev@dolphinscheduler.apache.org
> > > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; 主题: Re: [Feature] Support
> > > &gt; &amp;gt; SSH Task and Support dummy
> > > &gt; &amp;gt; &amp;amp;gt; task like airflow
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; The shell node is supports
> > > &gt; &amp;gt; remote calling, and get
> > > &gt; &amp;gt; &amp;amp;gt; the remote command result
> code.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; 在
> > > &gt; &amp;gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> > > &gt; &amp;gt; &amp;amp;gt; <mailto:peilongwu@qq.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > 写道：
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Dear ALL：
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Support Linux SSH
> > > &gt; &amp;gt; Task 支持 Linux SSH 任务
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> > > &gt; &amp;gt; &amp;amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker
> > > 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > > &gt; &amp;gt; &amp;amp;gt;
> > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; project, the workflow's tasks
> > > &gt; &amp;gt; &amp;amp;gt; want to execute shell scripts
> > where are in
> > > different server's
> > > &gt; &amp;gt; different
> > > &gt; &amp;gt; &amp;amp;gt; directory. When worker
> execute
> > these shell
> > > scripts, it must
> > > &gt; use the
> > > &gt; &amp;gt; same
> > > &gt; &amp;gt; &amp;amp;gt; user to login these server.
> > Also, the worker
> > > can get the
> > > &gt; executing
> > > &gt; &amp;gt; state of
> > > &gt; &amp;gt; &amp;amp;gt; these server. We can config
> > these server 's
> > > host,user and
> > > &gt; password.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; SSH Task is very
> > > &gt; &amp;gt; useful for most user SSH
> > > &gt; &amp;gt; &amp;amp;gt; 任务对大多数用户是非常有用的
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > > &gt; &amp;gt; &amp;amp;gt;
> > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > > Worker
> > > &gt; &amp;gt; &amp;amp;gt;
> 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; In
> > > &gt; &amp;gt; dolphinscheduler, the most executing tasks
> > > &gt; &amp;gt; &amp;amp;gt; are in different servers who
> > are not workers.
> > > These servers
> > > &gt; also have
> > > &gt; &amp;gt; their
> > > &gt; &amp;gt; &amp;amp;gt; different fixed services. We
> > just have to pass
> > > different
> > > &gt; parameters to
> > > &gt; &amp;gt; &amp;amp;gt; schedule these shell scripts
> > to execute.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; to execute ssh script Python
> > > &gt; &amp;gt; &amp;amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Python
> > > &gt; &amp;gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Python has a module
> > > &gt; &amp;gt; that can execute SSH Shell
> > > &gt; &amp;gt; &amp;amp;gt; script. It's paramiko.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Others 其他内容
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; I found this
> > > &gt; &amp;gt; described in previous feature, but
> > > &gt; &amp;gt; &amp;amp;gt; it was relatively simple.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Feature URL
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; 另外，我通过 Shell Task
> > > &gt; &amp;gt; &amp;amp;gt;
> > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; In addition, it is
> > > &gt; &amp;gt; very inconvenient for me to
> > > &gt; &amp;gt; &amp;amp;gt; perform remote tasks through
> > Shell Task. Here
> > > is my script.
> > > &gt; I don't
> > > &gt; &amp;gt; know if
> > > &gt; &amp;gt; &amp;amp;gt; there's a better way.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; sshpass -p
> > > &gt; &amp;gt; 'password' ssh user@host echo 'ssh
> > > &gt; &amp;gt; &amp;amp;gt; success' echo 'Hello World'
> > > -&amp;amp;amp;amp;gt;
> > > &gt; &amp;gt; /home/dolphinscheduler/test/hello.txt
> > > &gt; &amp;gt; &amp;amp;gt; echo 'end'
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Support dummy task
> > > &gt; &amp;gt; like airflow 支持像 Airflow
> > > &gt; &amp;gt; &amp;amp;gt; 中的虚拟任务
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > > &gt; &amp;gt; DAG 文件，DAG
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt;
> > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > > &gt; &amp;gt; Dummy
> > > &gt; &amp;gt; &amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> > Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; For example, in my
> > > &gt; &amp;gt; project, it has a productized
> > > &gt; &amp;gt; &amp;amp;gt; DAG file. The file contains
> > different modules,
> > > some of which
> > > &gt; are
> > > &gt; &amp;gt; &amp;amp;gt; interdependent and some of
> > which are not. When
> > > customers
> > > &gt; purchase
> > > &gt; &amp;gt; different
> > > &gt; &amp;gt; &amp;amp;gt; modules, we need to set some
> > tasks as dummy
> > > tasks, which
> > > &gt; some modules
> > > &gt; &amp;gt; are
> > > &gt; &amp;gt; &amp;amp;gt; not purchased and the
> > purchased module is not
> > > dependent.
> > > &gt; Because of
> > > &gt; &amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; setting, these dummy tasks
> are
> > actually not
> > > executed. The
> > > &gt; benefits of
> > > &gt; &amp;gt; this
> > > &gt; &amp;gt; &amp;amp;gt; setup are product unity and
> > diagram integrity.
> > > In airflow,
> > > &gt; these task
> > > &gt; &amp;gt; &amp;amp;gt; execute by dummy operator.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; ** Realize 实现方式**
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Dummy Task
> > > &gt; &amp;gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > > &gt; &amp;gt; &amp;amp;gt; dummy 时，实际的任务不执行，执行 Dummy
> Task。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; Dummy Task is easy
> > > &gt; &amp;gt; to realize, but it need to
> > > &gt; &amp;gt; &amp;amp;gt; use with other different
> > tasks. When the task's
> > > executed
> > > &gt; type is set
> > > &gt; &amp;gt; to
> > > &gt; &amp;gt; &amp;amp;gt; dummy type, the task are
> > executed as a dummy
> > > task and the
> > > &gt; real task
> > > &gt; &amp;gt; is not
> > > &gt; &amp;gt; &amp;amp;gt; executed.
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > > 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;amp;amp;gt; By the way，I
> > > &gt; &amp;gt; already realize these
> > two&amp;amp;amp;amp;nbsp;
> > > &gt; &amp;gt; &amp;amp;gt; features in my fork
> > > branch.&amp;amp;amp;amp;nbsp;Whether the
> > > &gt; follow-up
> > > &gt; &amp;gt; release can be
> > > &gt; &amp;gt; &amp;amp;gt; supported
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > >
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt;
> > > &gt;
> > >
> >
> <SSHClient.java&amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > > <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;gt;
>
Re: [Feature] Support SSH Task
Posted by lidong dai <da...@gmail.com>.
good idea, I think "Control the number of task's concurrent" may be have
some other methods, for now, we don't have this feature, but when I
communicate with the team of China Unicom， they provide a new solve
solution：
we can add a task concurrent page, you can set a variable value like
'ETL_JOB' =10 and so on, then give the task type add a option for choosing
ETL_JOB or other variable, and there may be many task also choose ETL_JOB
variable. when these tasks begin to running at the same time, the Master
Server will check whether a task begin to run or wait(if the running tasks
num = 10)
Best egards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <pe...@qq.com> 于2020年5月26日周二 下午2:34写道：
> There is only two ways that I can think to do about this.
>
>
> 1. Simple SSH connect pool
> 2. Control the number of task's concurrent
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Hemin Wen"<wenhemin@apache.org>;
> 发送时间: 2020年5月22日(星期五) 晚上7:29
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> What other friends think about this discuss
>
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月22日周五 上午10:20写道：
>
> > Your are right, it can be solved the problem by modifying the
> > number of server ssh connections.&nbsp;
> >
> > The real reason is that I want to control the&nbsp; number of SSH
> > connections and reuse the SSH connection, Because the remote server is
> > managed by other team.
> >
> > Otherwise, I think that we can also control the number of parallel
> tasks
> > in a workflow.
> >
> >
> > It is a good idea that we can use the encapsulated ssh connection to
> get
> > clear
> > exception information&nbsp;
> >
> >
> >
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> > 发送时间:&nbsp;2020年5月21日(星期四) 中午11:07
> > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task
> >
> >
> >
> > Thanks.
> >
> > About ssh connections pool,
> > 1. If the connection is not enough, Can it be solved by modifying the
> > number of server ssh connections.
> > 2. I think we have to look at whether a server's parallel ssh tasks
> will
> > reach the bottleneck of server connections.
> >
> > I think it is better to use the encapsulated ssh connection to get
> clear
> > exception information.
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen&nbsp; 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午11:29写道：
> >
> > &gt; Thanks for your advise and suggestions. It may has some
> exceptions if
> > I
> > &gt; implement SSH Task as you said. Now, let me explain my ideas.
> > &gt;
> > &gt;
> > &gt; e.g
> > &gt;
> > &gt;
> > &gt; My workflow DAG has 100 or more task, it's all ssh task. I
> > have&amp;nbsp; two
> > &gt; questions.
> > &gt;
> > &gt;
> > &gt; 1、We must limit SSH connection and reuse the SSH connections,
> > otherwise
> > &gt; the remote server will be refused.
> > &gt;
> > &gt;
> > &gt; 2、If we config all of these task, we have a lot of repeat
> work to do.
> > &gt;
> > &gt;
> > &gt; By the way, the way I'm currently verifying is I use local
> params to
> > &gt; config the remote server's host,user,password
> > &gt;
> > &gt;
> > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:&amp;nbsp;"lidong dai"<dailidong66@gmail.com
> &amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上10:56
> > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;gt;;
> > &gt;
> > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > &gt;
> > &gt;
> > &gt;
> > &gt; hi, I see you will Jsch, does his license compitable with
> apache v2
> > &gt; license, I see the following words "
> > &gt; JSch 0.0.* was released under the GNU LGPL license. Later,
> we have
> > switched
> > &gt; over to a BSD-style license.
> > &gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt .
> you know,
> > LGPL
> > &gt; license not compitable with apache license. but BSD is ok ,
> so the
> > version
> > &gt; of Jsch is key point
> > &gt;
> > &gt; second, Connection Pool is heavy resource, Please use
> carefully
> > &gt;
> > &gt; and I'm a little confused, why there must be add a new
> > Table,&amp;nbsp; other
> > &gt; task
> > &gt; types also no seperate table,&amp;nbsp; could you
> consider task_type
> > and
> > &gt; task_json
> > &gt; in t_ds_task_instance ?
> > &gt;
> > &gt;
> > &gt;
> > &gt; Best Regards
> > &gt; ---------------
> > &gt; DolphinScheduler(Incubator) PPMC
> > &gt; Lidong Dai 代立冬
> > &gt; dailidong66@gmail.com
> > &gt; ---------------
> > &gt;
> > &gt;
> > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午9:57写道：
> > &gt;
> > &gt; &amp;gt; I want to describe my idea.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; First, we must design a host table that's named
> > t_ds_ssh_host, e.g
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; - id
> > &gt; &amp;gt; - name
> > &gt; &amp;gt; - ip / host
> > &gt; &amp;gt; - user
> > &gt; &amp;gt; - password
> > &gt; &amp;gt; - max_connection
> > &gt; &amp;gt; - create_time
> > &gt; &amp;gt; - update_time
> > &gt; &amp;gt; - user_id
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Second, Shell can execute in worker server or
> remote host.
> > We can
> > &gt; choose a
> > &gt; &amp;gt; host in host's list.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Third, when workflow running, we must maintain
> the remote
> > host's
> > &gt; &amp;gt; connection pool.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Finally, when workflow finish, we must release
> the
> > connection pool.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Why do we have to maintain a connection pool.
> Because it is
> > very easy
> > &gt; to
> > &gt; &amp;gt; get exceptions when it's has many SSH task.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<
> dailidong66@gmail.com
> > &amp;amp;gt;;
> > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<
> dev@dolphinscheduler.apache.org
> > &amp;amp;gt;;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH
> Task
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; glad to hear that you will implement this
> feature
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Best Regards
> > &gt; &amp;gt; ---------------
> > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> > &gt; &amp;gt; Lidong Dai 代立冬
> > &gt; &amp;gt; dailidong66@gmail.com
> > &gt; &amp;gt; ---------------
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt;
> 于2020年5月20日周三 下午3:47写道：
> > &gt; &amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; My code is not perfect yet. I
> will write a
> > detailed design
> > &gt; document.
> > &gt; &amp;gt; Then
> > &gt; &amp;gt; &amp;amp;gt; I will realize this feature
> about our
> > discussion result.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> > &gt; &amp;gt; &amp;amp;gt;
> 发件人:&amp;amp;amp;nbsp;"wenhemin"<
> > whm_777@163.com&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> 发送时间:&amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > &gt; &amp;gt; &amp;amp;gt;
> 收件人:&amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> > &amp;amp;amp;gt;;"dev"<
> > &gt; &amp;gt; dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re:
> [Feature] Support SSH
> > Task and Support
> > &gt; dummy task
> > &gt; &amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Thanks for writing detailed
> documentation. I
> > think this is
> > &gt; also a
> > &gt; &amp;gt; missing
> > &gt; &amp;gt; &amp;amp;gt; feature of DS.
> > &gt; &amp;gt; &amp;amp;gt; About the extension point:
> > &gt; &amp;gt; &amp;amp;gt; 1.Can ssh tasks be merged into
> shell tasks.
> > Essentially,
> > &gt; they all
> > &gt; &amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; shell commands.
> > &gt; &amp;gt; &amp;amp;gt; 2.About dummy task, DS has the
> function of
> > disable nodes, I
> > &gt; do n’t
> > &gt; &amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; this requirement is met.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; The script from AirFlow to
> Dolphin is great.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 在
> 2020年5月18日，09:28，裴龙武 <
> > peilongwu@qq.com&amp;amp;amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; OK, 3Q!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, I
> will ensure that open
> > source can use.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second, I
> think we must
> > discuss deeply. I write
> > &gt; a more
> > &gt; &amp;gt; detailed
> > &gt; &amp;gt; &amp;amp;gt; document. You can check the
> attachment. I also
> > send the
> > &gt; document to
> > &gt; &amp;gt; &amp;amp;gt; DaiLidong.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> Third,&amp;amp;amp;nbsp; I'll
> > give you the error of
> > &gt; not using SSH
> > &gt; &amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> ------------------ 原始邮件
> > ------------------
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发件人:
> "wenhemin"<
> > whm_777@163.com&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发送时间:
> 2020年5月14日(星期四) 晚上7:26
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 收件人:
> "裴龙武"<peilongwu@qq.com
> > &amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题: Re:
> [Feature] Support SSH
> > Task and Support
> > &gt; dummy task
> > &gt; &amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Great!
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I think,
> Can ssh tasks be
> > merged into shell
> > &gt; tasks,&amp;amp;amp;nbsp;
> > &gt; &amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; script locally or remotely,
> Configure on the
> > front end.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About ssh
> connect pool, I did
> > not find it
> > &gt; necessary to use
> > &gt; &amp;gt; the
> > &gt; &amp;gt; &amp;amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; BTW, Look
> at the code to
> > introduce additional
> > &gt; jar packages,
> > &gt; &amp;gt; You also
> > &gt; &amp;gt; &amp;amp;gt; need to ensure that open
> source can use the
> > license of this
> > &gt; jar
> > &gt; &amp;gt; package.
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 在
> > 2020年5月14日，16:20，裴龙武 <
> > &gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 1. The
> > priority between these tasks
> > &gt; is also depended
> > &gt; &amp;gt; on the
> > &gt; &amp;gt; &amp;amp;gt; dolphin DAG define. When the
> front task is not
> > finished, it
> > &gt; not
> > &gt; &amp;gt; execute
> > &gt; &amp;gt; &amp;amp;gt; next task.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 2. I extend
> > ssh task. I also use
> > &gt; local params to
> > &gt; &amp;gt; config ssh host,
> > &gt; &amp;gt; &amp;amp;gt; user and password.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; E.g:
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; public static
> > AbstractTask
> > &gt; &amp;gt; newTask(TaskExecutionContext
> > &gt; &amp;gt; &amp;amp;gt; taskExecutionContext, Logger
> logger)
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; throws
> > &gt; &amp;gt; IllegalArgumentException {
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> Boolean
> > &gt; enable =
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> if
> > &gt; (enable != null
> > &gt; &amp;gt; &amp;amp;amp;amp;&amp;amp;amp;amp;
> enable == false ) {
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> DummyTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> }
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> switch
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case SHELL:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> ShellTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case
> > &gt; &amp;gt; PROCEDURE:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> ProcedureTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case SQL:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; SqlTask(taskExecutionContext,
> logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case MR:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> MapReduceTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case SPARK:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> SparkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case FLINK:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> FlinkTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case PYTHON:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> PythonTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case HTTP:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; HttpTask(taskExecutionContext,
> logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case DATAX:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> DataxTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case SQOOP:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt;
> SqoopTask(taskExecutionContext, logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; case SSH:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; return new
> > &gt; &amp;gt; &amp;amp;gt; SSHTask(taskExecutionContext,
> logger);
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; default:
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; logger.error("unsupport task
> > &gt; &amp;gt; &amp;amp;gt; type: {}",
> taskExecutionContext.getTaskType());
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; throw new
> > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException("not
> support task
> > type");
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> }
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; }
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 3. I am not
> > sure that it supports
> > &gt; window or not.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > ------------------ 原始邮件
> > &gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> > "wenhemin"<whm_777@163.com
> > &gt; <mailto:
> > &gt; &amp;gt; whm_777@163.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> > 2020年5月14日(星期四) 下午3:46
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"<
> > peilongwu@qq.com
> > &gt; <mailto:peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> > [Feature] Support SSH Task
> > &gt; and Support dummy
> > &gt; &amp;gt; task like
> > &gt; &amp;gt; &amp;amp;gt; airflow
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; Sorry, My
> > previous description is
> > &gt; not very clear.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; I want to ask
> > some questions:
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 1.How to
> > control the priority
> > &gt; between ssh tasks?
> > &gt; &amp;gt; There may be
> > &gt; &amp;gt; &amp;amp;gt; some ssh tasks that have been
> waiting for
> > execution.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 2.I understand
> > what you want to
> > &gt; solve is the problem
> > &gt; &amp;gt; of executing
> > &gt; &amp;gt; &amp;amp;gt; remote ssh scripts in batches.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> So, not
> > &gt; sure how to use this
> > &gt; &amp;gt; function.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt; 3.I don't know
> > if this supports
> > &gt; windows system.
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 在
> 2020年5月13日，20:56，裴龙武 <
> > &gt; peilongwu@qq.com
> > &gt; &amp;gt; <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; I use
> spin lock. Here
> > &gt; is my code. Of course
> > &gt; &amp;gt; , it's not
> > &gt; &amp;gt; &amp;amp;gt; perfect. I just do a test. To
> my surprise, it
> > is the result
> > &gt; of the
> > &gt; &amp;gt; &amp;amp;gt; execution is the same as the
> AirFlow
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt;
> 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > &gt; &amp;gt; &amp;amp;gt; 相同的结果。
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> ------------------ 原始邮件
> > &gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> "whm_777"<
> > &gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> 2020年5月13日(星期三)
> > &gt; 晚上7:21
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 收件人:
> "裴龙武"<
> > &gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> [Feature]
> > &gt; Support SSH Task and
> > &gt; &amp;gt; Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; You can
> modify the
> > &gt; maximum number of linux
> > &gt; &amp;gt; ssh connections.
> > &gt; &amp;gt; &amp;amp;gt;
> > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; If use
> ssh connection
> > &gt; pool, How to control
> > &gt; &amp;gt; the priority of
> > &gt; &amp;gt; &amp;amp;gt; ssh?
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> 在
> > &gt; 2020年5月13日，18:01，裴龙武 <
> > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> First 3Q，
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; I&amp;amp;amp;nbsp; use more than 100 task
> > &gt; &amp;gt; node. But SSH connections
> > &gt; &amp;gt; &amp;amp;gt; are limited.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; <330BE502@F7F80E73.76C5BB5E.jpg
> > &gt; &amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; ------------------ 原始邮件
> > &gt; &amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> 发件人:
> > &gt; "whm_777"<whm_777@163.com
> > &gt; &amp;gt; <mailto:whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> 发送时间:
> > &gt; 2020年5月13日(星期三) 下午5:50
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> 收件人: "裴龙武"<
> > &gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> 主题: Re:
> > &gt; [Feature] Support SSH Task
> > &gt; &amp;gt; and Support dummy task
> > &gt; &amp;gt; &amp;amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> E.g.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; rtn_code=`ssh -o
> > &gt; &amp;gt; ServerAliveInterval=60 -p xxxx
> > &gt; &amp;gt; &amp;amp;gt; root@xxx.xxx.xxx.xxx <mailto:
> > root@xxx.xxx.xxx.xxx&amp;amp;amp;gt;
> > &gt; ‘shell
> > &gt; &amp;gt; &amp;amp;gt; command&amp;amp;amp;nbsp;
> > &amp;amp;amp;gt;/dev/null
> > &gt; 2&amp;amp;amp;gt;&amp;amp;amp;amp;1; echo $?'`
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> if [
> > &gt; "$rtn_code" -eq 0 ]; then
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; echo "成功"
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; exit 0
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> else
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; echo "失败"
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> > &gt; &amp;gt; exit 1
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> fi
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> Batch shell
> > &gt; command is not supported.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> Multiple
> > &gt; servers can be split into
> > &gt; &amp;gt; multiple task nodes.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 在 2020年5月13日，17:40，裴龙武 <
> > &gt; &amp;gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; Could you give me a
> > &gt; &amp;gt; example，3Q. 能否给我一个例子，谢谢！
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; By the way, I have more than
> > &gt; &amp;gt; 100 tasks in one DAG.
> > &gt; &amp;gt; &amp;amp;gt; These tasks connect two other
> server to
> > execute. So SSH
> > &gt; tasks must
> > &gt; &amp;gt; have
> > &gt; &amp;gt; &amp;amp;gt; pool to manager. Now I use
> JSch and realize a
> > simple pool.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> > &gt; &amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > &gt; &amp;gt; &amp;amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用
> JSch，并实现了一个简单的连接池。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; ------------------ 原始邮件
> > &gt; &amp;gt; ------------------
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 发件人: "wenhemin"<
> > &gt; &amp;gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 收件人: "dev"<
> > &gt; &amp;gt; dev@dolphinscheduler.apache.org <mailto:
> > &gt; &amp;gt; &amp;amp;gt;
> dev@dolphinscheduler.apache.org
> > &amp;amp;amp;gt;&amp;amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; 主题: Re: [Feature] Support
> > &gt; &amp;gt; SSH Task and Support dummy
> > &gt; &amp;gt; &amp;amp;gt; task like airflow
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; The shell node is supports
> > &gt; &amp;gt; remote calling, and get
> > &gt; &amp;gt; &amp;amp;gt; the remote command result code.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; 在
> > &gt; &amp;gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt; <mailto:peilongwu@qq.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Dear ALL：
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Support Linux SSH
> > &gt; &amp;gt; Task 支持 Linux SSH 任务
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> > &gt; &amp;gt; &amp;amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker
> > 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > &gt; &amp;gt; &amp;amp;gt;
> 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; project, the workflow's tasks
> > &gt; &amp;gt; &amp;amp;gt; want to execute shell scripts
> where are in
> > different server's
> > &gt; &amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; directory. When worker execute
> these shell
> > scripts, it must
> > &gt; use the
> > &gt; &amp;gt; same
> > &gt; &amp;gt; &amp;amp;gt; user to login these server.
> Also, the worker
> > can get the
> > &gt; executing
> > &gt; &amp;gt; state of
> > &gt; &amp;gt; &amp;amp;gt; these server. We can config
> these server 's
> > host,user and
> > &gt; password.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; SSH Task is very
> > &gt; &amp;gt; useful for most user SSH
> > &gt; &amp;gt; &amp;amp;gt; 任务对大多数用户是非常有用的
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> > &gt; &amp;gt; &amp;amp;gt;
> 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> > Worker
> > &gt; &amp;gt; &amp;amp;gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; In
> > &gt; &amp;gt; dolphinscheduler, the most executing tasks
> > &gt; &amp;gt; &amp;amp;gt; are in different servers who
> are not workers.
> > These servers
> > &gt; also have
> > &gt; &amp;gt; their
> > &gt; &amp;gt; &amp;amp;gt; different fixed services. We
> just have to pass
> > different
> > &gt; parameters to
> > &gt; &amp;gt; &amp;amp;gt; schedule these shell scripts
> to execute.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; to execute ssh script Python
> > &gt; &amp;gt; &amp;amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Python
> > &gt; &amp;gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Python has a module
> > &gt; &amp;gt; that can execute SSH Shell
> > &gt; &amp;gt; &amp;amp;gt; script. It's paramiko.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Others 其他内容
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; I found this
> > &gt; &amp;gt; described in previous feature, but
> > &gt; &amp;gt; &amp;amp;gt; it was relatively simple.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Feature URL
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; 另外，我通过 Shell Task
> > &gt; &amp;gt; &amp;amp;gt;
> 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; In addition, it is
> > &gt; &amp;gt; very inconvenient for me to
> > &gt; &amp;gt; &amp;amp;gt; perform remote tasks through
> Shell Task. Here
> > is my script.
> > &gt; I don't
> > &gt; &amp;gt; know if
> > &gt; &amp;gt; &amp;amp;gt; there's a better way.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; sshpass -p
> > &gt; &amp;gt; 'password' ssh user@host echo 'ssh
> > &gt; &amp;gt; &amp;amp;gt; success' echo 'Hello World'
> > -&amp;amp;amp;amp;gt;
> > &gt; &amp;gt; /home/dolphinscheduler/test/hello.txt
> > &gt; &amp;gt; &amp;amp;gt; echo 'end'
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Support dummy task
> > &gt; &amp;gt; like airflow 支持像 Airflow
> > &gt; &amp;gt; &amp;amp;gt; 中的虚拟任务
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> > &gt; &amp;gt; DAG 文件，DAG
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt;
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > &gt; &amp;gt; Dummy
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; For example, in my
> > &gt; &amp;gt; project, it has a productized
> > &gt; &amp;gt; &amp;amp;gt; DAG file. The file contains
> different modules,
> > some of which
> > &gt; are
> > &gt; &amp;gt; &amp;amp;gt; interdependent and some of
> which are not. When
> > customers
> > &gt; purchase
> > &gt; &amp;gt; different
> > &gt; &amp;gt; &amp;amp;gt; modules, we need to set some
> tasks as dummy
> > tasks, which
> > &gt; some modules
> > &gt; &amp;gt; are
> > &gt; &amp;gt; &amp;amp;gt; not purchased and the
> purchased module is not
> > dependent.
> > &gt; Because of
> > &gt; &amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; setting, these dummy tasks are
> actually not
> > executed. The
> > &gt; benefits of
> > &gt; &amp;gt; this
> > &gt; &amp;gt; &amp;amp;gt; setup are product unity and
> diagram integrity.
> > In airflow,
> > &gt; these task
> > &gt; &amp;gt; &amp;amp;gt; execute by dummy operator.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; ** Realize 实现方式**
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Dummy Task
> > &gt; &amp;gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > &gt; &amp;gt; &amp;amp;gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; Dummy Task is easy
> > &gt; &amp;gt; to realize, but it need to
> > &gt; &amp;gt; &amp;amp;gt; use with other different
> tasks. When the task's
> > executed
> > &gt; type is set
> > &gt; &amp;gt; to
> > &gt; &amp;gt; &amp;amp;gt; dummy type, the task are
> executed as a dummy
> > task and the
> > &gt; real task
> > &gt; &amp;gt; is not
> > &gt; &amp;gt; &amp;amp;gt; executed.
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;amp;amp;gt; By the way，I
> > &gt; &amp;gt; already realize these
> two&amp;amp;amp;amp;nbsp;
> > &gt; &amp;gt; &amp;amp;gt; features in my fork
> > branch.&amp;amp;amp;amp;nbsp;Whether the
> > &gt; follow-up
> > &gt; &amp;gt; release can be
> > &gt; &amp;gt; &amp;amp;gt; supported
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> >
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> <SSHClient.java&amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> > <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;gt;
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
There is only two ways that I can think to do about this.
1. Simple SSH connect pool
2. Control the number of task's concurrent
------------------ 原始邮件 ------------------
发件人: "Hemin Wen"<wenhemin@apache.org>;
发送时间: 2020年5月22日(星期五) 晚上7:29
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
What other friends think about this discuss
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
裴龙武 <peilongwu@qq.com> 于2020年5月22日周五 上午10:20写道：
> Your are right, it can be solved the problem by modifying the
> number of server ssh connections.&nbsp;
>
> The real reason is that I want to control the&nbsp; number of SSH
> connections and reuse the SSH connection, Because the remote server is
> managed by other team.
>
> Otherwise, I think that we can also control the number of parallel tasks
> in a workflow.
>
>
> It is a good idea that we can use the encapsulated ssh connection to get
> clear
> exception information&nbsp;
>
>
>
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"Hemin Wen"<wenhemin@apache.org&gt;;
> 发送时间:&nbsp;2020年5月21日(星期四) 中午11:07
> 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
>
> 主题:&nbsp;Re: [Feature] Support SSH Task
>
>
>
> Thanks.
>
> About ssh connections pool,
> 1. If the connection is not enough, Can it be solved by modifying the
> number of server ssh connections.
> 2. I think we have to look at whether a server's parallel ssh tasks will
> reach the bottleneck of server connections.
>
> I think it is better to use the encapsulated ssh connection to get clear
> exception information.
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen&nbsp; 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午11:29写道：
>
> &gt; Thanks for your advise and suggestions. It may has some exceptions if
> I
> &gt; implement SSH Task as you said. Now, let me explain my ideas.
> &gt;
> &gt;
> &gt; e.g
> &gt;
> &gt;
> &gt; My workflow DAG has 100 or more task, it's all ssh task. I
> have&amp;nbsp; two
> &gt; questions.
> &gt;
> &gt;
> &gt; 1、We must limit SSH connection and reuse the SSH connections,
> otherwise
> &gt; the remote server will be refused.
> &gt;
> &gt;
> &gt; 2、If we config all of these task, we have a lot of repeat work to do.
> &gt;
> &gt;
> &gt; By the way, the way I'm currently verifying is I use local params to
> &gt; config the remote server's host,user,password
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:&amp;nbsp;"lidong dai"<dailidong66@gmail.com&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上10:56
> &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org&amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> &gt;
> &gt;
> &gt;
> &gt; hi, I see you will Jsch, does his license compitable with apache v2
> &gt; license, I see the following words "
> &gt; JSch 0.0.* was released under the GNU LGPL license. Later, we have
> switched
> &gt; over to a BSD-style license.
> &gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know,
> LGPL
> &gt; license not compitable with apache license. but BSD is ok , so the
> version
> &gt; of Jsch is key point
> &gt;
> &gt; second, Connection Pool is heavy resource, Please use carefully
> &gt;
> &gt; and I'm a little confused, why there must be add a new
> Table,&amp;nbsp; other
> &gt; task
> &gt; types also no seperate table,&amp;nbsp; could you consider task_type
> and
> &gt; task_json
> &gt; in t_ds_task_instance ?
> &gt;
> &gt;
> &gt;
> &gt; Best Regards
> &gt; ---------------
> &gt; DolphinScheduler(Incubator) PPMC
> &gt; Lidong Dai 代立冬
> &gt; dailidong66@gmail.com
> &gt; ---------------
> &gt;
> &gt;
> &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午9:57写道：
> &gt;
> &gt; &amp;gt; I want to describe my idea.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; First, we must design a host table that's named
> t_ds_ssh_host, e.g
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; - id
> &gt; &amp;gt; - name
> &gt; &amp;gt; - ip / host
> &gt; &amp;gt; - user
> &gt; &amp;gt; - password
> &gt; &amp;gt; - max_connection
> &gt; &amp;gt; - create_time
> &gt; &amp;gt; - update_time
> &gt; &amp;gt; - user_id
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Second, Shell can execute in worker server or remote host.
> We can
> &gt; choose a
> &gt; &amp;gt; host in host's list.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Third, when workflow running, we must maintain the remote
> host's
> &gt; &amp;gt; connection pool.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Finally, when workflow finish, we must release the
> connection pool.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Why do we have to maintain a connection pool. Because it is
> very easy
> &gt; to
> &gt; &amp;gt; get exceptions when it's has many SSH task.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"<dailidong66@gmail.com
> &amp;amp;gt;;
> &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12
> &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;amp;gt;;
> &gt; &amp;gt;
> &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH Task
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; glad to hear that you will implement this feature
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Best Regards
> &gt; &amp;gt; ---------------
> &gt; &amp;gt; DolphinScheduler(Incubator) PPMC
> &gt; &amp;gt; Lidong Dai 代立冬
> &gt; &amp;gt; dailidong66@gmail.com
> &gt; &amp;gt; ---------------
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; 裴龙武 <peilongwu@qq.com&amp;amp;gt; 于2020年5月20日周三 下午3:47写道：
> &gt; &amp;gt;
> &gt; &amp;gt; &amp;amp;gt; My code is not perfect yet. I will write a
> detailed design
> &gt; document.
> &gt; &amp;gt; Then
> &gt; &amp;gt; &amp;amp;gt; I will realize this feature about our
> discussion result.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------
> &gt; &amp;gt; &amp;amp;gt; 发件人:&amp;amp;amp;nbsp;"wenhemin"<
> whm_777@163.com&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; 发送时间:&amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> &gt; &amp;gt; &amp;amp;gt; 收件人:&amp;amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> &amp;amp;amp;gt;;"dev"<
> &gt; &amp;gt; dev@dolphinscheduler.apache.org
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re: [Feature] Support SSH
> Task and Support
> &gt; dummy task
> &gt; &amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Thanks for writing detailed documentation. I
> think this is
> &gt; also a
> &gt; &amp;gt; missing
> &gt; &amp;gt; &amp;amp;gt; feature of DS.
> &gt; &amp;gt; &amp;amp;gt; About the extension point:
> &gt; &amp;gt; &amp;amp;gt; 1.Can ssh tasks be merged into shell tasks.
> Essentially,
> &gt; they all
> &gt; &amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; shell commands.
> &gt; &amp;gt; &amp;amp;gt; 2.About dummy task, DS has the function of
> disable nodes, I
> &gt; do n’t
> &gt; &amp;gt; know if
> &gt; &amp;gt; &amp;amp;gt; this requirement is met.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; The script from AirFlow to Dolphin is great.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 在 2020年5月18日，09:28，裴龙武 <
> peilongwu@qq.com&amp;amp;amp;gt;
> &gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; OK, 3Q!
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, I will ensure that open
> source can use.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second, I think we must
> discuss deeply. I write
> &gt; a more
> &gt; &amp;gt; detailed
> &gt; &amp;gt; &amp;amp;gt; document. You can check the attachment. I also
> send the
> &gt; document to
> &gt; &amp;gt; &amp;amp;gt; DaiLidong.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Third,&amp;amp;amp;nbsp; I'll
> give you the error of
> &gt; not using SSH
> &gt; &amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ------------------ 原始邮件
> ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发件人: "wenhemin"<
> whm_777@163.com&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 收件人: "裴龙武"<peilongwu@qq.com
> &amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题: Re: [Feature] Support SSH
> Task and Support
> &gt; dummy task
> &gt; &amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Great!
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I think, Can ssh tasks be
> merged into shell
> &gt; tasks,&amp;amp;amp;nbsp;
> &gt; &amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; script locally or remotely, Configure on the
> front end.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About ssh connect pool, I did
> not find it
> &gt; necessary to use
> &gt; &amp;gt; the
> &gt; &amp;gt; &amp;amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; BTW, Look at the code to
> introduce additional
> &gt; jar packages,
> &gt; &amp;gt; You also
> &gt; &amp;gt; &amp;amp;gt; need to ensure that open source can use the
> license of this
> &gt; jar
> &gt; &amp;gt; package.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 在
> 2020年5月14日，16:20，裴龙武 <
> &gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 1. The
> priority between these tasks
> &gt; is also depended
> &gt; &amp;gt; on the
> &gt; &amp;gt; &amp;amp;gt; dolphin DAG define. When the front task is not
> finished, it
> &gt; not
> &gt; &amp;gt; execute
> &gt; &amp;gt; &amp;amp;gt; next task.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 2. I extend
> ssh task. I also use
> &gt; local params to
> &gt; &amp;gt; config ssh host,
> &gt; &amp;gt; &amp;amp;gt; user and password.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; E.g:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; public static
> AbstractTask
> &gt; &amp;gt; newTask(TaskExecutionContext
> &gt; &amp;gt; &amp;amp;gt; taskExecutionContext, Logger logger)
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; throws
> &gt; &amp;gt; IllegalArgumentException {
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; Boolean
> &gt; enable =
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; if
> &gt; (enable != null
> &gt; &amp;gt; &amp;amp;amp;amp;&amp;amp;amp;amp; enable == false ) {
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; &amp;amp;gt; DummyTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; }
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; switch
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case SHELL:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; ShellTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case
> &gt; &amp;gt; PROCEDURE:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; ProcedureTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case SQL:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; SqlTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case MR:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; MapReduceTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case SPARK:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; SparkTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case FLINK:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; FlinkTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case PYTHON:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; PythonTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case HTTP:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; HttpTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case DATAX:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; DataxTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case SQOOP:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; SqoopTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; case SSH:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; return new
> &gt; &amp;gt; &amp;amp;gt; SSHTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; default:
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; logger.error("unsupport task
> &gt; &amp;gt; &amp;amp;gt; type: {}", taskExecutionContext.getTaskType());
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; throw new
> &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException("not support task
> type");
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; }
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; }
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 3. I am not
> sure that it supports
> &gt; window or not.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> ------------------ 原始邮件
> &gt; ------------------
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> "wenhemin"<whm_777@163.com
> &gt; <mailto:
> &gt; &amp;gt; whm_777@163.com&amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> 2020年5月14日(星期四) 下午3:46
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com
> &gt; <mailto:peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> [Feature] Support SSH Task
> &gt; and Support dummy
> &gt; &amp;gt; task like
> &gt; &amp;gt; &amp;amp;gt; airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; Sorry, My
> previous description is
> &gt; not very clear.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; I want to ask
> some questions:
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 1.How to
> control the priority
> &gt; between ssh tasks?
> &gt; &amp;gt; There may be
> &gt; &amp;gt; &amp;amp;gt; some ssh tasks that have been waiting for
> execution.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 2.I understand
> what you want to
> &gt; solve is the problem
> &gt; &amp;gt; of executing
> &gt; &amp;gt; &amp;amp;gt; remote ssh scripts in batches.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; So, not
> &gt; sure how to use this
> &gt; &amp;gt; function.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 3.I don't know
> if this supports
> &gt; windows system.
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 在 2020年5月13日，20:56，裴龙武 <
> &gt; peilongwu@qq.com
> &gt; &amp;gt; <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; I use spin lock. Here
> &gt; is my code. Of course
> &gt; &amp;gt; , it's not
> &gt; &amp;gt; &amp;amp;gt; perfect. I just do a test. To my surprise, it
> is the result
> &gt; of the
> &gt; &amp;gt; &amp;amp;gt; execution is the same as the AirFlow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> &gt; &amp;gt; &amp;amp;gt; 相同的结果。
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ------------------ 原始邮件
> &gt; ------------------
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发件人: "whm_777"<
> &gt; whm_777@163.com <mailto:
> &gt; &amp;gt; whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间: 2020年5月13日(星期三)
> &gt; 晚上7:21
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"<
> &gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re: [Feature]
> &gt; Support SSH Task and
> &gt; &amp;gt; Support dummy task
> &gt; &amp;gt; &amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; You can modify the
> &gt; maximum number of linux
> &gt; &amp;gt; ssh connections.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; If use ssh connection
> &gt; pool, How to control
> &gt; &amp;gt; the priority of
> &gt; &amp;gt; &amp;amp;gt; ssh?
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 在
> &gt; 2020年5月13日，18:01，裴龙武 <
> &gt; &amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; First 3Q，
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; I&amp;amp;amp;nbsp; use more than 100 task
> &gt; &amp;gt; node. But SSH connections
> &gt; &amp;gt; &amp;amp;gt; are limited.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; <330BE502@F7F80E73.76C5BB5E.jpg
> &gt; &amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; ------------------ 原始邮件
> &gt; &amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发件人:
> &gt; "whm_777"<whm_777@163.com
> &gt; &amp;gt; <mailto:whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间:
> &gt; 2020年5月13日(星期三) 下午5:50
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"<
> &gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re:
> &gt; [Feature] Support SSH Task
> &gt; &amp;gt; and Support dummy task
> &gt; &amp;gt; &amp;amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; E.g.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; rtn_code=`ssh -o
> &gt; &amp;gt; ServerAliveInterval=60 -p xxxx
> &gt; &amp;gt; &amp;amp;gt; root@xxx.xxx.xxx.xxx <mailto:
> root@xxx.xxx.xxx.xxx&amp;amp;amp;gt;
> &gt; ‘shell
> &gt; &amp;gt; &amp;amp;gt; command&amp;amp;amp;nbsp;
> &amp;amp;amp;gt;/dev/null
> &gt; 2&amp;amp;amp;gt;&amp;amp;amp;amp;1; echo $?'`
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; if [
> &gt; "$rtn_code" -eq 0 ]; then
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; echo "成功"
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; exit 0
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; else
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; echo "失败"
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;
> &gt; &amp;gt; exit 1
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; fi
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; Batch shell
> &gt; command is not supported.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; Multiple
> &gt; servers can be split into
> &gt; &amp;gt; multiple task nodes.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 在 2020年5月13日，17:40，裴龙武 <
> &gt; &amp;gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; Could you give me a
> &gt; &amp;gt; example，3Q. 能否给我一个例子，谢谢！
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; By the way, I have more than
> &gt; &amp;gt; 100 tasks in one DAG.
> &gt; &amp;gt; &amp;amp;gt; These tasks connect two other server to
> execute. So SSH
> &gt; tasks must
> &gt; &amp;gt; have
> &gt; &amp;gt; &amp;amp;gt; pool to manager. Now I use JSch and realize a
> simple pool.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> &gt; &amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> &gt; &amp;gt; &amp;amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; ------------------ 原始邮件
> &gt; &amp;gt; ------------------
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 发件人: "wenhemin"<
> &gt; &amp;gt; whm_777@163.com <mailto:
> &gt; &amp;gt; &amp;amp;gt; whm_777@163.com
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 收件人: "dev"<
> &gt; &amp;gt; dev@dolphinscheduler.apache.org <mailto:
> &gt; &amp;gt; &amp;amp;gt; dev@dolphinscheduler.apache.org
> &amp;amp;amp;gt;&amp;amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; 主题: Re: [Feature] Support
> &gt; &amp;gt; SSH Task and Support dummy
> &gt; &amp;gt; &amp;amp;gt; task like airflow
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; The shell node is supports
> &gt; &amp;gt; remote calling, and get
> &gt; &amp;gt; &amp;amp;gt; the remote command result code.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; 在
> &gt; &amp;gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt; <mailto:peilongwu@qq.com&amp;amp;amp;gt;&amp;amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Dear ALL：
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Support Linux SSH
> &gt; &amp;gt; Task 支持 Linux SSH 任务
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> &gt; &amp;gt; &amp;amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker
> 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> &gt; &amp;gt; &amp;amp;gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; For example, in my
> &gt; &amp;gt; project, the workflow's tasks
> &gt; &amp;gt; &amp;amp;gt; want to execute shell scripts where are in
> different server's
> &gt; &amp;gt; different
> &gt; &amp;gt; &amp;amp;gt; directory. When worker execute these shell
> scripts, it must
> &gt; use the
> &gt; &amp;gt; same
> &gt; &amp;gt; &amp;amp;gt; user to login these server. Also, the worker
> can get the
> &gt; executing
> &gt; &amp;gt; state of
> &gt; &amp;gt; &amp;amp;gt; these server. We can config these server 's
> host,user and
> &gt; password.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; SSH Task is very
> &gt; &amp;gt; useful for most user SSH
> &gt; &amp;gt; &amp;amp;gt; 任务对大多数用户是非常有用的
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; 分布式调度任务所执行的 Shell
> &gt; &amp;gt; &amp;amp;gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> Worker
> &gt; &amp;gt; &amp;amp;gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; In
> &gt; &amp;gt; dolphinscheduler, the most executing tasks
> &gt; &amp;gt; &amp;amp;gt; are in different servers who are not workers.
> These servers
> &gt; also have
> &gt; &amp;gt; their
> &gt; &amp;gt; &amp;amp;gt; different fixed services. We just have to pass
> different
> &gt; parameters to
> &gt; &amp;gt; &amp;amp;gt; schedule these shell scripts to execute.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Python has a module
> &gt; &amp;gt; to execute ssh script Python
> &gt; &amp;gt; &amp;amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Python
> &gt; &amp;gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Python has a module
> &gt; &amp;gt; that can execute SSH Shell
> &gt; &amp;gt; &amp;amp;gt; script. It's paramiko.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Others 其他内容
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; I found this
> &gt; &amp;gt; described in previous feature, but
> &gt; &amp;gt; &amp;amp;gt; it was relatively simple.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Feature URL
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; 另外，我通过 Shell Task
> &gt; &amp;gt; &amp;amp;gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; In addition, it is
> &gt; &amp;gt; very inconvenient for me to
> &gt; &amp;gt; &amp;amp;gt; perform remote tasks through Shell Task. Here
> is my script.
> &gt; I don't
> &gt; &amp;gt; know if
> &gt; &amp;gt; &amp;amp;gt; there's a better way.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; sshpass -p
> &gt; &amp;gt; 'password' ssh user@host echo 'ssh
> &gt; &amp;gt; &amp;amp;gt; success' echo 'Hello World'
> -&amp;amp;amp;amp;gt;
> &gt; &amp;gt; /home/dolphinscheduler/test/hello.txt
> &gt; &amp;gt; &amp;amp;gt; echo 'end'
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Support dummy task
> &gt; &amp;gt; like airflow 支持像 Airflow
> &gt; &amp;gt; &amp;amp;gt; 中的虚拟任务
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; 场景描述：项目中，有已经产品化的
> &gt; &amp;gt; DAG 文件，DAG
> &gt; &amp;gt; &amp;amp;gt;
> &gt; 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> &gt; &amp;gt; Dummy
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; For example, in my
> &gt; &amp;gt; project, it has a productized
> &gt; &amp;gt; &amp;amp;gt; DAG file. The file contains different modules,
> some of which
> &gt; are
> &gt; &amp;gt; &amp;amp;gt; interdependent and some of which are not. When
> customers
> &gt; purchase
> &gt; &amp;gt; different
> &gt; &amp;gt; &amp;amp;gt; modules, we need to set some tasks as dummy
> tasks, which
> &gt; some modules
> &gt; &amp;gt; are
> &gt; &amp;gt; &amp;amp;gt; not purchased and the purchased module is not
> dependent.
> &gt; Because of
> &gt; &amp;gt; this
> &gt; &amp;gt; &amp;amp;gt; setting, these dummy tasks are actually not
> executed. The
> &gt; benefits of
> &gt; &amp;gt; this
> &gt; &amp;gt; &amp;amp;gt; setup are product unity and diagram integrity.
> In airflow,
> &gt; these task
> &gt; &amp;gt; &amp;amp;gt; execute by dummy operator.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; ** Realize 实现方式**
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Dummy Task
> &gt; &amp;gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> &gt; &amp;gt; &amp;amp;gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; Dummy Task is easy
> &gt; &amp;gt; to realize, but it need to
> &gt; &amp;gt; &amp;amp;gt; use with other different tasks. When the task's
> executed
> &gt; type is set
> &gt; &amp;gt; to
> &gt; &amp;gt; &amp;amp;gt; dummy type, the task are executed as a dummy
> task and the
> &gt; real task
> &gt; &amp;gt; is not
> &gt; &amp;gt; &amp;amp;gt; executed.
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;amp;amp;gt; By the way，I
> &gt; &amp;gt; already realize these two&amp;amp;amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;gt; features in my fork
> branch.&amp;amp;amp;amp;nbsp;Whether the
> &gt; follow-up
> &gt; &amp;gt; release can be
> &gt; &amp;gt; &amp;amp;gt; supported
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> <SSHClient.java&amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;
> <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;gt;
Re: [Feature] Support SSH Task
Posted by Hemin Wen <we...@apache.org>.
What other friends think about this discuss
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
裴龙武 <pe...@qq.com> 于2020年5月22日周五 上午10:20写道：
> Your are right, it can be solved the problem by modifying the
> number of server ssh connections.
>
> The real reason is that I want to control the number of SSH
> connections and reuse the SSH connection, Because the remote server is
> managed by other team.
>
> Otherwise, I think that we can also control the number of parallel tasks
> in a workflow.
>
>
> It is a good idea that we can use the encapsulated ssh connection to get
> clear
> exception information
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Hemin Wen"<wenhemin@apache.org>;
> 发送时间: 2020年5月21日(星期四) 中午11:07
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> Thanks.
>
> About ssh connections pool,
> 1. If the connection is not enough, Can it be solved by modifying the
> number of server ssh connections.
> 2. I think we have to look at whether a server's parallel ssh tasks will
> reach the bottleneck of server connections.
>
> I think it is better to use the encapsulated ssh connection to get clear
> exception information.
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午11:29写道：
>
> > Thanks for your advise and suggestions. It may has some exceptions if
> I
> > implement SSH Task as you said. Now, let me explain my ideas.
> >
> >
> > e.g
> >
> >
> > My workflow DAG has 100 or more task, it's all ssh task. I
> have&nbsp; two
> > questions.
> >
> >
> > 1、We must limit SSH connection and reuse the SSH connections,
> otherwise
> > the remote server will be refused.
> >
> >
> > 2、If we config all of these task, we have a lot of repeat work to do.
> >
> >
> > By the way, the way I'm currently verifying is I use local params to
> > config the remote server's host,user,password
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> > 发送时间:&nbsp;2020年5月20日(星期三) 晚上10:56
> > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task
> >
> >
> >
> > hi, I see you will Jsch, does his license compitable with apache v2
> > license, I see the following words "
> > JSch 0.0.* was released under the GNU LGPL license. Later, we have
> switched
> > over to a BSD-style license.
> > " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know,
> LGPL
> > license not compitable with apache license. but BSD is ok , so the
> version
> > of Jsch is key point
> >
> > second, Connection Pool is heavy resource, Please use carefully
> >
> > and I'm a little confused, why there must be add a new
> Table,&nbsp; other
> > task
> > types also no seperate table,&nbsp; could you consider task_type
> and
> > task_json
> > in t_ds_task_instance ?
> >
> >
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) PPMC
> > Lidong Dai 代立冬
> > dailidong66@gmail.com
> > ---------------
> >
> >
> > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午9:57写道：
> >
> > &gt; I want to describe my idea.
> > &gt;
> > &gt;
> > &gt; First, we must design a host table that's named
> t_ds_ssh_host, e.g
> > &gt;
> > &gt;
> > &gt; - id
> > &gt; - name
> > &gt; - ip / host
> > &gt; - user
> > &gt; - password
> > &gt; - max_connection
> > &gt; - create_time
> > &gt; - update_time
> > &gt; - user_id
> > &gt;
> > &gt;
> > &gt; Second, Shell can execute in worker server or remote host.
> We can
> > choose a
> > &gt; host in host's list.
> > &gt;
> > &gt;
> > &gt; Third, when workflow running, we must maintain the remote
> host's
> > &gt; connection pool.
> > &gt;
> > &gt;
> > &gt; Finally, when workflow finish, we must release the
> connection pool.
> > &gt;
> > &gt;
> > &gt; Why do we have to maintain a connection pool. Because it is
> very easy
> > to
> > &gt; get exceptions when it's has many SSH task.
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:&amp;nbsp;"lidong dai"<dailidong66@gmail.com
> &amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上7:12
> > &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org
> &amp;gt;;
> > &gt;
> > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> > &gt;
> > &gt;
> > &gt;
> > &gt; glad to hear that you will implement this feature
> > &gt;
> > &gt;
> > &gt;
> > &gt; Best Regards
> > &gt; ---------------
> > &gt; DolphinScheduler(Incubator) PPMC
> > &gt; Lidong Dai 代立冬
> > &gt; dailidong66@gmail.com
> > &gt; ---------------
> > &gt;
> > &gt;
> > &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午3:47写道：
> > &gt;
> > &gt; &amp;gt; My code is not perfect yet. I will write a
> detailed design
> > document.
> > &gt; Then
> > &gt; &amp;gt; I will realize this feature about our
> discussion result.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> >
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"wenhemin"<
> whm_777@163.com&amp;amp;gt;;
> > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"裴龙武"<peilongwu@qq.com
> &amp;amp;gt;;"dev"<
> > &gt; dev@dolphinscheduler.apache.org
> > &gt; &amp;gt; &amp;amp;gt;;
> > &gt; &amp;gt;
> > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH
> Task and Support
> > dummy task
> > &gt; like airflow
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; Thanks for writing detailed documentation. I
> think this is
> > also a
> > &gt; missing
> > &gt; &amp;gt; feature of DS.
> > &gt; &amp;gt; About the extension point:
> > &gt; &amp;gt; 1.Can ssh tasks be merged into shell tasks.
> Essentially,
> > they all
> > &gt; execute
> > &gt; &amp;gt; shell commands.
> > &gt; &amp;gt; 2.About dummy task, DS has the function of
> disable nodes, I
> > do n’t
> > &gt; know if
> > &gt; &amp;gt; this requirement is met.
> > &gt; &amp;gt;
> > &gt; &amp;gt; The script from AirFlow to Dolphin is great.
> > &gt; &amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; 在 2020年5月18日，09:28，裴龙武 <
> peilongwu@qq.com&amp;amp;gt;
> > 写道：
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; OK, 3Q!
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; First, I will ensure that open
> source can use.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Second, I think we must
> discuss deeply. I write
> > a more
> > &gt; detailed
> > &gt; &amp;gt; document. You can check the attachment. I also
> send the
> > document to
> > &gt; &amp;gt; DaiLidong.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Third,&amp;amp;nbsp; I'll
> give you the error of
> > not using SSH
> > &gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件
> ------------------
> > &gt; &amp;gt; &amp;amp;gt; 发件人: "wenhemin"<
> whm_777@163.com&amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> > &gt; &amp;gt; &amp;amp;gt; 收件人: "裴龙武"<peilongwu@qq.com
> &amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt; 主题: Re: [Feature] Support SSH
> Task and Support
> > dummy task
> > &gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; Great!
> > &gt; &amp;gt; &amp;amp;gt; I think, Can ssh tasks be
> merged into shell
> > tasks,&amp;amp;nbsp;
> > &gt; execute
> > &gt; &amp;gt; script locally or remotely, Configure on the
> front end.
> > &gt; &amp;gt; &amp;amp;gt; About ssh connect pool, I did
> not find it
> > necessary to use
> > &gt; the
> > &gt; &amp;gt; connection pool.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt; BTW, Look at the code to
> introduce additional
> > jar packages,
> > &gt; You also
> > &gt; &amp;gt; need to ensure that open source can use the
> license of this
> > jar
> > &gt; package.
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 在
> 2020年5月14日，16:20，裴龙武 <
> > peilongwu@qq.com <mailto:
> > &gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; 写道：
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 1. The
> priority between these tasks
> > is also depended
> > &gt; on the
> > &gt; &amp;gt; dolphin DAG define. When the front task is not
> finished, it
> > not
> > &gt; execute
> > &gt; &amp;gt; next task.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 2. I extend
> ssh task. I also use
> > local params to
> > &gt; config ssh host,
> > &gt; &amp;gt; user and password.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; E.g:
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; public static
> AbstractTask
> > &gt; newTask(TaskExecutionContext
> > &gt; &amp;gt; taskExecutionContext, Logger logger)
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > throws
> > &gt; IllegalArgumentException {
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; Boolean
> > enable =
> > &gt; &amp;gt;
> > &gt;
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; if
> > (enable != null
> > &gt; &amp;amp;amp;&amp;amp;amp; enable == false ) {
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > return new
> > &gt; &amp;gt; DummyTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; }
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; switch
> > &gt; &amp;gt;
> > &gt;
> >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case SHELL:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; ShellTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case
> > &gt; PROCEDURE:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; ProcedureTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case SQL:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; SqlTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case MR:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; MapReduceTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case SPARK:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; SparkTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case FLINK:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; FlinkTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case PYTHON:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; PythonTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case HTTP:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; HttpTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case DATAX:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; DataxTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case SQOOP:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; SqoopTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > case SSH:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; return new
> > &gt; &amp;gt; SSHTask(taskExecutionContext, logger);
> > &gt; &amp;gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > default:
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; logger.error("unsupport task
> > &gt; &amp;gt; type: {}", taskExecutionContext.getTaskType());
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; throw new
> > &gt; &amp;gt; IllegalArgumentException("not support task
> type");
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; }
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; }
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 3. I am not
> sure that it supports
> > window or not.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> ------------------ 原始邮件
> > ------------------
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 发件人:
> "wenhemin"<whm_777@163.com
> > <mailto:
> > &gt; whm_777@163.com&amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 发送时间:
> 2020年5月14日(星期四) 下午3:46
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com
> > <mailto:peilongwu@qq.com
> > &gt; &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 主题: Re:
> [Feature] Support SSH Task
> > and Support dummy
> > &gt; task like
> > &gt; &amp;gt; airflow
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; Sorry, My
> previous description is
> > not very clear.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; I want to ask
> some questions:
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 1.How to
> control the priority
> > between ssh tasks?
> > &gt; There may be
> > &gt; &amp;gt; some ssh tasks that have been waiting for
> execution.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 2.I understand
> what you want to
> > solve is the problem
> > &gt; of executing
> > &gt; &amp;gt; remote ssh scripts in batches.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; So, not
> > sure how to use this
> > &gt; function.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 3.I don't know
> if this supports
> > windows system.
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 在 2020年5月13日，20:56，裴龙武 <
> > peilongwu@qq.com
> > &gt; <mailto:
> > &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> 写道：
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; I use spin lock. Here
> > is my code. Of course
> > &gt; , it's not
> > &gt; &amp;gt; perfect. I just do a test. To my surprise, it
> is the result
> > of the
> > &gt; &amp;gt; execution is the same as the AirFlow
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > &gt; &amp;gt; 相同的结果。
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; ------------------ 原始邮件
> > ------------------
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发件人: "whm_777"<
> > whm_777@163.com <mailto:
> > &gt; whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发送时间: 2020年5月13日(星期三)
> > 晚上7:21
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<
> > peilongwu@qq.com <mailto:
> > &gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 主题: Re: [Feature]
> > Support SSH Task and
> > &gt; Support dummy task
> > &gt; &amp;gt; like airflow
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; You can modify the
> > maximum number of linux
> > &gt; ssh connections.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; If use ssh connection
> > pool, How to control
> > &gt; the priority of
> > &gt; &amp;gt; ssh?
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 在
> > 2020年5月13日，18:01，裴龙武 <
> > &gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> 写道：
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; First 3Q，
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > I&amp;amp;nbsp; use more than 100 task
> > &gt; node. But SSH connections
> > &gt; &amp;gt; are limited.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> > &gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > <330BE502@F7F80E73.76C5BB5E.jpg
> > &gt; &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > ------------------ 原始邮件
> > &gt; ------------------
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发件人:
> > "whm_777"<whm_777@163.com
> > &gt; <mailto:whm_777@163.com
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发送时间:
> > 2020年5月13日(星期三) 下午5:50
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<
> > peilongwu@qq.com <mailto:
> > &gt; peilongwu@qq.com
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 主题: Re:
> > [Feature] Support SSH Task
> > &gt; and Support dummy task
> > &gt; &amp;gt; like airflow
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; E.g.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > rtn_code=`ssh -o
> > &gt; ServerAliveInterval=60 -p xxxx
> > &gt; &amp;gt; root@xxx.xxx.xxx.xxx <mailto:
> root@xxx.xxx.xxx.xxx&amp;amp;gt;
> > ‘shell
> > &gt; &amp;gt; command&amp;amp;nbsp;
> &amp;amp;gt;/dev/null
> > 2&amp;amp;gt;&amp;amp;amp;1; echo $?'`
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; if [
> > "$rtn_code" -eq 0 ]; then
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; echo "成功"
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; exit 0
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; else
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; echo "失败"
> > &gt; &amp;gt;
> > &gt;
> >
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> > &gt; exit 1
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; fi
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; Batch shell
> > command is not supported.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; Multiple
> > servers can be split into
> > &gt; multiple task nodes.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 在 2020年5月13日，17:40，裴龙武 <
> > &gt; peilongwu@qq.com <mailto:
> > &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> 写道：
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > Could you give me a
> > &gt; example，3Q. 能否给我一个例子，谢谢！
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > By the way, I have more than
> > &gt; 100 tasks in one DAG.
> > &gt; &amp;gt; These tasks connect two other server to
> execute. So SSH
> > tasks must
> > &gt; have
> > &gt; &amp;gt; pool to manager. Now I use JSch and realize a
> simple pool.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 顺带说一下，在我的实际场景中，我有100多个 SSH
> > &gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> > &gt; &amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > ------------------ 原始邮件
> > &gt; ------------------
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 发件人: "wenhemin"<
> > &gt; whm_777@163.com <mailto:
> > &gt; &amp;gt; whm_777@163.com
> &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 收件人: "dev"<
> > &gt; dev@dolphinscheduler.apache.org <mailto:
> > &gt; &amp;gt; dev@dolphinscheduler.apache.org
> &amp;amp;gt;&amp;amp;gt;;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > 主题: Re: [Feature] Support
> > &gt; SSH Task and Support dummy
> > &gt; &amp;gt; task like airflow
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > The shell node is supports
> > &gt; remote calling, and get
> > &gt; &amp;gt; the remote command result code.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; 在
> > &gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> > &gt; &amp;gt; <mailto:peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> 写道：
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Dear ALL：
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Support Linux SSH
> > &gt; Task 支持 Linux SSH 任务
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> > &gt; &amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker
> 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > &gt; &amp;gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; For example, in my
> > &gt; project, the workflow's tasks
> > &gt; &amp;gt; want to execute shell scripts where are in
> different server's
> > &gt; different
> > &gt; &amp;gt; directory. When worker execute these shell
> scripts, it must
> > use the
> > &gt; same
> > &gt; &amp;gt; user to login these server. Also, the worker
> can get the
> > executing
> > &gt; state of
> > &gt; &amp;gt; these server. We can config these server 's
> host,user and
> > password.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; SSH Task is very
> > &gt; useful for most user SSH
> > &gt; &amp;gt; 任务对大多数用户是非常有用的
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; 分布式调度任务所执行的 Shell
> > &gt; &amp;gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要
> Worker
> > &gt; &amp;gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; In
> > &gt; dolphinscheduler, the most executing tasks
> > &gt; &amp;gt; are in different servers who are not workers.
> These servers
> > also have
> > &gt; their
> > &gt; &amp;gt; different fixed services. We just have to pass
> different
> > parameters to
> > &gt; &amp;gt; schedule these shell scripts to execute.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Python has a module
> > &gt; to execute ssh script Python
> > &gt; &amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Python
> > &gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Python has a module
> > &gt; that can execute SSH Shell
> > &gt; &amp;gt; script. It's paramiko.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Others 其他内容
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; I found this
> > &gt; described in previous feature, but
> > &gt; &amp;gt; it was relatively simple.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Feature URL
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; 另外，我通过 Shell Task
> > &gt; &amp;gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; In addition, it is
> > &gt; very inconvenient for me to
> > &gt; &amp;gt; perform remote tasks through Shell Task. Here
> is my script.
> > I don't
> > &gt; know if
> > &gt; &amp;gt; there's a better way.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; sshpass -p
> > &gt; 'password' ssh user@host echo 'ssh
> > &gt; &amp;gt; success' echo 'Hello World'
> -&amp;amp;amp;gt;
> > &gt; /home/dolphinscheduler/test/hello.txt
> > &gt; &amp;gt; echo 'end'
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Support dummy task
> > &gt; like airflow 支持像 Airflow
> > &gt; &amp;gt; 中的虚拟任务
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; 场景描述：项目中，有已经产品化的
> > &gt; DAG 文件，DAG
> > &gt; &amp;gt;
> > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > &gt; Dummy
> > &gt; &amp;gt;
> > &gt;
> >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; For example, in my
> > &gt; project, it has a productized
> > &gt; &amp;gt; DAG file. The file contains different modules,
> some of which
> > are
> > &gt; &amp;gt; interdependent and some of which are not. When
> customers
> > purchase
> > &gt; different
> > &gt; &amp;gt; modules, we need to set some tasks as dummy
> tasks, which
> > some modules
> > &gt; are
> > &gt; &amp;gt; not purchased and the purchased module is not
> dependent.
> > Because of
> > &gt; this
> > &gt; &amp;gt; setting, these dummy tasks are actually not
> executed. The
> > benefits of
> > &gt; this
> > &gt; &amp;gt; setup are product unity and diagram integrity.
> In airflow,
> > these task
> > &gt; &amp;gt; execute by dummy operator.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; ** Realize 实现方式**
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Dummy Task
> > &gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > &gt; &amp;gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; Dummy Task is easy
> > &gt; to realize, but it need to
> > &gt; &amp;gt; use with other different tasks. When the task's
> executed
> > type is set
> > &gt; to
> > &gt; &amp;gt; dummy type, the task are executed as a dummy
> task and the
> > real task
> > &gt; is not
> > &gt; &amp;gt; executed.
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &amp;amp;gt; By the way，I
> > &gt; already realize these two&amp;amp;amp;nbsp;
> > &gt; &amp;gt; features in my fork
> branch.&amp;amp;amp;nbsp;Whether the
> > follow-up
> > &gt; release can be
> > &gt; &amp;gt; supported
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> > &gt;
> >
> <SSHClient.java&amp;amp;gt;<SSHPool.java&amp;amp;gt;<SSHTask.java&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> > &gt; &amp;gt; &amp;amp;gt;
> <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;gt;
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
Your are right, it can be solved the problem by modifying the
number of server ssh connections.
The real reason is that I want to control the number of SSH connections and reuse the SSH connection, Because the remote server is managed by other team.
Otherwise, I think that we can also control the number of parallel tasks in a workflow.
It is a good idea that we can use the encapsulated ssh connection to get clear
exception information
------------------ 原始邮件 ------------------
发件人: "Hemin Wen"<wenhemin@apache.org>;
发送时间: 2020年5月21日(星期四) 中午11:07
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
Thanks.
About ssh connections pool,
1. If the connection is not enough, Can it be solved by modifying the
number of server ssh connections.
2. I think we have to look at whether a server's parallel ssh tasks will
reach the bottleneck of server connections.
I think it is better to use the encapsulated ssh connection to get clear
exception information.
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午11:29写道：
> Thanks for your advise and suggestions. It may has some exceptions if I
> implement SSH Task as you said. Now, let me explain my ideas.
>
>
> e.g
>
>
> My workflow DAG has 100 or more task, it's all ssh task. I have&nbsp; two
> questions.
>
>
> 1、We must limit SSH connection and reuse the SSH connections, otherwise
> the remote server will be refused.
>
>
> 2、If we config all of these task, we have a lot of repeat work to do.
>
>
> By the way, the way I'm currently verifying is I use local params to
> config the remote server's host,user,password
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> 发送时间:&nbsp;2020年5月20日(星期三) 晚上10:56
> 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
>
> 主题:&nbsp;Re: [Feature] Support SSH Task
>
>
>
> hi, I see you will Jsch, does his license compitable with apache v2
> license, I see the following words "
> JSch 0.0.* was released under the GNU LGPL license. Later, we have switched
> over to a BSD-style license.
> " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, LGPL
> license not compitable with apache license. but BSD is ok , so the version
> of Jsch is key point
>
> second, Connection Pool is heavy resource, Please use carefully
>
> and I'm a little confused, why there must be add a new Table,&nbsp; other
> task
> types also no seperate table,&nbsp; could you consider task_type and
> task_json
> in t_ds_task_instance ?
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午9:57写道：
>
> &gt; I want to describe my idea.
> &gt;
> &gt;
> &gt; First, we must design a host table that's named t_ds_ssh_host, e.g
> &gt;
> &gt;
> &gt; - id
> &gt; - name
> &gt; - ip / host
> &gt; - user
> &gt; - password
> &gt; - max_connection
> &gt; - create_time
> &gt; - update_time
> &gt; - user_id
> &gt;
> &gt;
> &gt; Second, Shell can execute in worker server or remote host. We can
> choose a
> &gt; host in host's list.
> &gt;
> &gt;
> &gt; Third, when workflow running, we must maintain the remote host's
> &gt; connection pool.
> &gt;
> &gt;
> &gt; Finally, when workflow finish, we must release the connection pool.
> &gt;
> &gt;
> &gt; Why do we have to maintain a connection pool. Because it is very easy
> to
> &gt; get exceptions when it's has many SSH task.
> &gt;
> &gt;
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:&amp;nbsp;"lidong dai"<dailidong66@gmail.com&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上7:12
> &gt; 收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org&amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task
> &gt;
> &gt;
> &gt;
> &gt; glad to hear that you will implement this feature
> &gt;
> &gt;
> &gt;
> &gt; Best Regards
> &gt; ---------------
> &gt; DolphinScheduler(Incubator) PPMC
> &gt; Lidong Dai 代立冬
> &gt; dailidong66@gmail.com
> &gt; ---------------
> &gt;
> &gt;
> &gt; 裴龙武 <peilongwu@qq.com&amp;gt; 于2020年5月20日周三 下午3:47写道：
> &gt;
> &gt; &amp;gt; My code is not perfect yet. I will write a detailed design
> document.
> &gt; Then
> &gt; &amp;gt; I will realize this feature about our discussion result.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> &gt; &amp;gt; 发件人:&amp;amp;nbsp;"wenhemin"<whm_777@163.com&amp;amp;gt;;
> &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50
> &gt; &amp;gt; 收件人:&amp;amp;nbsp;"裴龙武"<peilongwu@qq.com&amp;amp;gt;;"dev"<
> &gt; dev@dolphinscheduler.apache.org
> &gt; &amp;gt; &amp;amp;gt;;
> &gt; &amp;gt;
> &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH Task and Support
> dummy task
> &gt; like airflow
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Thanks for writing detailed documentation. I think this is
> also a
> &gt; missing
> &gt; &amp;gt; feature of DS.
> &gt; &amp;gt; About the extension point:
> &gt; &amp;gt; 1.Can ssh tasks be merged into shell tasks. Essentially,
> they all
> &gt; execute
> &gt; &amp;gt; shell commands.
> &gt; &amp;gt; 2.About dummy task, DS has the function of disable nodes, I
> do n’t
> &gt; know if
> &gt; &amp;gt; this requirement is met.
> &gt; &amp;gt;
> &gt; &amp;gt; The script from AirFlow to Dolphin is great.
> &gt; &amp;gt;
> &gt; &amp;gt; &amp;amp;gt; 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com&amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; OK, 3Q!
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; First, I will ensure that open source can use.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Second, I think we must discuss deeply. I write
> a more
> &gt; detailed
> &gt; &amp;gt; document. You can check the attachment. I also send the
> document to
> &gt; &amp;gt; DaiLidong.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Third,&amp;amp;nbsp; I'll give you the error of
> not using SSH
> &gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; ------------------ 原始邮件 ------------------
> &gt; &amp;gt; &amp;amp;gt; 发件人: "wenhemin"<whm_777@163.com&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> &gt; &amp;gt; &amp;amp;gt; 收件人: "裴龙武"<peilongwu@qq.com&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt; 主题: Re: [Feature] Support SSH Task and Support
> dummy task
> &gt; like airflow
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; Great!
> &gt; &amp;gt; &amp;amp;gt; I think, Can ssh tasks be merged into shell
> tasks,&amp;amp;nbsp;
> &gt; execute
> &gt; &amp;gt; script locally or remotely, Configure on the front end.
> &gt; &amp;gt; &amp;amp;gt; About ssh connect pool, I did not find it
> necessary to use
> &gt; the
> &gt; &amp;gt; connection pool.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; BTW, Look at the code to introduce additional
> jar packages,
> &gt; You also
> &gt; &amp;gt; need to ensure that open source can use the license of this
> jar
> &gt; package.
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 在 2020年5月14日，16:20，裴龙武 <
> peilongwu@qq.com <mailto:
> &gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 1. The priority between these tasks
> is also depended
> &gt; on the
> &gt; &amp;gt; dolphin DAG define. When the front task is not finished, it
> not
> &gt; execute
> &gt; &amp;gt; next task.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 2. I extend ssh task. I also use
> local params to
> &gt; config ssh host,
> &gt; &amp;gt; user and password.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; E.g:
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; public static AbstractTask
> &gt; newTask(TaskExecutionContext
> &gt; &amp;gt; taskExecutionContext, Logger logger)
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> throws
> &gt; IllegalArgumentException {
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; Boolean
> enable =
> &gt; &amp;gt;
> &gt;
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; if
> (enable != null
> &gt; &amp;amp;amp;&amp;amp;amp; enable == false ) {
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> return new
> &gt; &amp;gt; DummyTask(taskExecutionContext, logger);
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; }
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; switch
> &gt; &amp;gt;
> &gt;
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case SHELL:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; ShellTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case
> &gt; PROCEDURE:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; ProcedureTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case SQL:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; SqlTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case MR:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; MapReduceTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case SPARK:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; SparkTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case FLINK:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; FlinkTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case PYTHON:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; PythonTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case HTTP:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; HttpTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case DATAX:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; DataxTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case SQOOP:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; SqoopTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> case SSH:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; return new
> &gt; &amp;gt; SSHTask(taskExecutionContext, logger);
> &gt; &amp;gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> default:
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; logger.error("unsupport task
> &gt; &amp;gt; type: {}", taskExecutionContext.getTaskType());
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; throw new
> &gt; &amp;gt; IllegalArgumentException("not support task type");
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; }
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; }
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 3. I am not sure that it supports
> window or not.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; ------------------ 原始邮件
> ------------------
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 发件人: "wenhemin"<whm_777@163.com
> <mailto:
> &gt; whm_777@163.com&amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 发送时间: 2020年5月14日(星期四) 下午3:46
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<peilongwu@qq.com
> <mailto:peilongwu@qq.com
> &gt; &amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 主题: Re: [Feature] Support SSH Task
> and Support dummy
> &gt; task like
> &gt; &amp;gt; airflow
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; Sorry, My previous description is
> not very clear.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; I want to ask some questions:
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 1.How to control the priority
> between ssh tasks?
> &gt; There may be
> &gt; &amp;gt; some ssh tasks that have been waiting for execution.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 2.I understand what you want to
> solve is the problem
> &gt; of executing
> &gt; &amp;gt; remote ssh scripts in batches.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp; So, not
> sure how to use this
> &gt; function.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt; 3.I don't know if this supports
> windows system.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 在 2020年5月13日，20:56，裴龙武 <
> peilongwu@qq.com
> &gt; <mailto:
> &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; I use spin lock. Here
> is my code. Of course
> &gt; , it's not
> &gt; &amp;gt; perfect. I just do a test. To my surprise, it is the result
> of the
> &gt; &amp;gt; execution is the same as the AirFlow
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> &gt; &amp;gt; 相同的结果。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; ------------------ 原始邮件
> ------------------
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发件人: "whm_777"<
> whm_777@163.com <mailto:
> &gt; whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发送时间: 2020年5月13日(星期三)
> 晚上7:21
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com <mailto:
> &gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 主题: Re: [Feature]
> Support SSH Task and
> &gt; Support dummy task
> &gt; &amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; You can modify the
> maximum number of linux
> &gt; ssh connections.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt; If use ssh connection
> pool, How to control
> &gt; the priority of
> &gt; &amp;gt; ssh?
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 在
> 2020年5月13日，18:01，裴龙武 <
> &gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; First 3Q，
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> I&amp;amp;nbsp; use more than 100 task
> &gt; node. But SSH connections
> &gt; &amp;gt; are limited.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> <330BE502@F7F80E73.76C5BB5E.jpg
> &gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> ------------------ 原始邮件
> &gt; ------------------
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发件人:
> "whm_777"<whm_777@163.com
> &gt; <mailto:whm_777@163.com
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 发送时间:
> 2020年5月13日(星期三) 下午5:50
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com <mailto:
> &gt; peilongwu@qq.com
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; 主题: Re:
> [Feature] Support SSH Task
> &gt; and Support dummy task
> &gt; &amp;gt; like airflow
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; E.g.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> rtn_code=`ssh -o
> &gt; ServerAliveInterval=60 -p xxxx
> &gt; &amp;gt; root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx&amp;amp;gt;
> ‘shell
> &gt; &amp;gt; command&amp;amp;nbsp; &amp;amp;gt;/dev/null
> 2&amp;amp;gt;&amp;amp;amp;1; echo $?'`
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; if [
> "$rtn_code" -eq 0 ]; then
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; echo "成功"
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; exit 0
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; else
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; echo "失败"
> &gt; &amp;gt;
> &gt;
> &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;&amp;amp;nbsp;
> &gt; exit 1
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; fi
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; Batch shell
> command is not supported.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; Multiple
> servers can be split into
> &gt; multiple task nodes.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 在 2020年5月13日，17:40，裴龙武 <
> &gt; peilongwu@qq.com <mailto:
> &gt; &amp;gt; peilongwu@qq.com&amp;amp;gt;&amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> Could you give me a
> &gt; example，3Q. 能否给我一个例子，谢谢！
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> By the way, I have more than
> &gt; 100 tasks in one DAG.
> &gt; &amp;gt; These tasks connect two other server to execute. So SSH
> tasks must
> &gt; have
> &gt; &amp;gt; pool to manager. Now I use JSch and realize a simple pool.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 顺带说一下，在我的实际场景中，我有100多个 SSH
> &gt; 任务，这些任务连接两台任务服务器进行任务执行。所以
> &gt; &amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> ------------------ 原始邮件
> &gt; ------------------
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 发件人: "wenhemin"<
> &gt; whm_777@163.com <mailto:
> &gt; &amp;gt; whm_777@163.com&amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 发送时间: 2020年5月13日(星期三) 下午5:24
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 收件人: "dev"<
> &gt; dev@dolphinscheduler.apache.org <mailto:
> &gt; &amp;gt; dev@dolphinscheduler.apache.org&amp;amp;gt;&amp;amp;gt;;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> 主题: Re: [Feature] Support
> &gt; SSH Task and Support dummy
> &gt; &amp;gt; task like airflow
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> The shell node is supports
> &gt; remote calling, and get
> &gt; &amp;gt; the remote command result code.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; 在
> &gt; 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> &gt; &amp;gt; <mailto:peilongwu@qq.com&amp;amp;gt;&amp;amp;gt; 写道：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Dear ALL：
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Support Linux SSH
> &gt; Task 支持 Linux SSH 任务
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> &gt; &amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> &gt; &amp;gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; For example, in my
> &gt; project, the workflow's tasks
> &gt; &amp;gt; want to execute shell scripts where are in different server's
> &gt; different
> &gt; &amp;gt; directory. When worker execute these shell scripts, it must
> use the
> &gt; same
> &gt; &amp;gt; user to login these server. Also, the worker can get the
> executing
> &gt; state of
> &gt; &amp;gt; these server. We can config these server 's host,user and
> password.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; SSH Task is very
> &gt; useful for most user SSH
> &gt; &amp;gt; 任务对大多数用户是非常有用的
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; 分布式调度任务所执行的 Shell
> &gt; &amp;gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> &gt; &amp;gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; In
> &gt; dolphinscheduler, the most executing tasks
> &gt; &amp;gt; are in different servers who are not workers. These servers
> also have
> &gt; their
> &gt; &amp;gt; different fixed services. We just have to pass different
> parameters to
> &gt; &amp;gt; schedule these shell scripts to execute.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Python has a module
> &gt; to execute ssh script Python
> &gt; &amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Python
> &gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Python has a module
> &gt; that can execute SSH Shell
> &gt; &amp;gt; script. It's paramiko.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Others 其他内容
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; I found this
> &gt; described in previous feature, but
> &gt; &amp;gt; it was relatively simple.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Feature URL
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; 另外，我通过 Shell Task
> &gt; &amp;gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; In addition, it is
> &gt; very inconvenient for me to
> &gt; &amp;gt; perform remote tasks through Shell Task. Here is my script.
> I don't
> &gt; know if
> &gt; &amp;gt; there's a better way.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; sshpass -p
> &gt; 'password' ssh user@host echo 'ssh
> &gt; &amp;gt; success' echo 'Hello World' -&amp;amp;amp;gt;
> &gt; /home/dolphinscheduler/test/hello.txt
> &gt; &amp;gt; echo 'end'
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Support dummy task
> &gt; like airflow 支持像 Airflow
> &gt; &amp;gt; 中的虚拟任务
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; 场景描述：项目中，有已经产品化的
> &gt; DAG 文件，DAG
> &gt; &amp;gt;
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> &gt; Dummy
> &gt; &amp;gt;
> &gt;
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; For example, in my
> &gt; project, it has a productized
> &gt; &amp;gt; DAG file. The file contains different modules, some of which
> are
> &gt; &amp;gt; interdependent and some of which are not. When customers
> purchase
> &gt; different
> &gt; &amp;gt; modules, we need to set some tasks as dummy tasks, which
> some modules
> &gt; are
> &gt; &amp;gt; not purchased and the purchased module is not dependent.
> Because of
> &gt; this
> &gt; &amp;gt; setting, these dummy tasks are actually not executed. The
> benefits of
> &gt; this
> &gt; &amp;gt; setup are product unity and diagram integrity. In airflow,
> these task
> &gt; &amp;gt; execute by dummy operator.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; ** Realize 实现方式**
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Dummy Task
> &gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> &gt; &amp;gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; Dummy Task is easy
> &gt; to realize, but it need to
> &gt; &amp;gt; use with other different tasks. When the task's executed
> type is set
> &gt; to
> &gt; &amp;gt; dummy type, the task are executed as a dummy task and the
> real task
> &gt; is not
> &gt; &amp;gt; executed.
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &amp;amp;gt; By the way，I
> &gt; already realize these two&amp;amp;amp;nbsp;
> &gt; &amp;gt; features in my fork branch.&amp;amp;amp;nbsp;Whether the
> follow-up
> &gt; release can be
> &gt; &amp;gt; supported
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;&amp;amp;gt;
> &gt;
> <SSHClient.java&amp;amp;gt;<SSHPool.java&amp;amp;gt;<SSHTask.java&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt; &amp;amp;gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;gt;
Re: [Feature] Support SSH Task
Posted by Hemin Wen <we...@apache.org>.
Thanks.
About ssh connections pool,
1. If the connection is not enough, Can it be solved by modifying the
number of server ssh connections.
2. I think we have to look at whether a server's parallel ssh tasks will
reach the bottleneck of server connections.
I think it is better to use the encapsulated ssh connection to get clear
exception information.
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
裴龙武 <pe...@qq.com> 于2020年5月20日周三 下午11:29写道：
> Thanks for your advise and suggestions. It may has some exceptions if I
> implement SSH Task as you said. Now, let me explain my ideas.
>
>
> e.g
>
>
> My workflow DAG has 100 or more task, it's all ssh task. I have two
> questions.
>
>
> 1、We must limit SSH connection and reuse the SSH connections, otherwise
> the remote server will be refused.
>
>
> 2、If we config all of these task, we have a lot of repeat work to do.
>
>
> By the way, the way I'm currently verifying is I use local params to
> config the remote server's host,user,password
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "lidong dai"<dailidong66@gmail.com>;
> 发送时间: 2020年5月20日(星期三) 晚上10:56
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> hi, I see you will Jsch, does his license compitable with apache v2
> license, I see the following words "
> JSch 0.0.* was released under the GNU LGPL license. Later, we have switched
> over to a BSD-style license.
> " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, LGPL
> license not compitable with apache license. but BSD is ok , so the version
> of Jsch is key point
>
> second, Connection Pool is heavy resource, Please use carefully
>
> and I'm a little confused, why there must be add a new Table, other
> task
> types also no seperate table, could you consider task_type and
> task_json
> in t_ds_task_instance ?
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午9:57写道：
>
> > I want to describe my idea.
> >
> >
> > First, we must design a host table that's named t_ds_ssh_host, e.g
> >
> >
> > - id
> > - name
> > - ip / host
> > - user
> > - password
> > - max_connection
> > - create_time
> > - update_time
> > - user_id
> >
> >
> > Second, Shell can execute in worker server or remote host. We can
> choose a
> > host in host's list.
> >
> >
> > Third, when workflow running, we must maintain the remote host's
> > connection pool.
> >
> >
> > Finally, when workflow finish, we must release the connection pool.
> >
> >
> > Why do we have to maintain a connection pool. Because it is very easy
> to
> > get exceptions when it's has many SSH task.
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> > 发送时间:&nbsp;2020年5月20日(星期三) 晚上7:12
> > 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task
> >
> >
> >
> > glad to hear that you will implement this feature
> >
> >
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) PPMC
> > Lidong Dai 代立冬
> > dailidong66@gmail.com
> > ---------------
> >
> >
> > 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午3:47写道：
> >
> > &gt; My code is not perfect yet. I will write a detailed design
> document.
> > Then
> > &gt; I will realize this feature about our discussion result.
> > &gt;
> > &gt;
> > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:&amp;nbsp;"wenhemin"<whm_777@163.com&amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2020年5月18日(星期一) 晚上7:50
> > &gt; 收件人:&amp;nbsp;"裴龙武"<peilongwu@qq.com&amp;gt;;"dev"<
> > dev@dolphinscheduler.apache.org
> > &gt; &amp;gt;;
> > &gt;
> > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task and Support
> dummy task
> > like airflow
> > &gt;
> > &gt;
> > &gt;
> > &gt; Thanks for writing detailed documentation. I think this is
> also a
> > missing
> > &gt; feature of DS.
> > &gt; About the extension point:
> > &gt; 1.Can ssh tasks be merged into shell tasks. Essentially,
> they all
> > execute
> > &gt; shell commands.
> > &gt; 2.About dummy task, DS has the function of disable nodes, I
> do n’t
> > know if
> > &gt; this requirement is met.
> > &gt;
> > &gt; The script from AirFlow to Dolphin is great.
> > &gt;
> > &gt; &amp;gt; 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com&amp;gt;
> 写道：
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; OK, 3Q!
> > &gt; &amp;gt;
> > &gt; &amp;gt; First, I will ensure that open source can use.
> > &gt; &amp;gt;
> > &gt; &amp;gt; Second, I think we must discuss deeply. I write
> a more
> > detailed
> > &gt; document. You can check the attachment. I also send the
> document to
> > &gt; DaiLidong.
> > &gt; &amp;gt;
> > &gt; &amp;gt; Third,&amp;nbsp; I'll give you the error of
> not using SSH
> > connection pool.
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; ------------------ 原始邮件 ------------------
> > &gt; &amp;gt; 发件人: "wenhemin"<whm_777@163.com&amp;gt;;
> > &gt; &amp;gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> > &gt; &amp;gt; 收件人: "裴龙武"<peilongwu@qq.com&amp;gt;;
> > &gt; &amp;gt; 主题: Re: [Feature] Support SSH Task and Support
> dummy task
> > like airflow
> > &gt; &amp;gt;
> > &gt; &amp;gt; Great!
> > &gt; &amp;gt; I think, Can ssh tasks be merged into shell
> tasks,&amp;nbsp;
> > execute
> > &gt; script locally or remotely, Configure on the front end.
> > &gt; &amp;gt; About ssh connect pool, I did not find it
> necessary to use
> > the
> > &gt; connection pool.
> > &gt; &amp;gt;
> > &gt; &amp;gt; BTW, Look at the code to introduce additional
> jar packages,
> > You also
> > &gt; need to ensure that open source can use the license of this
> jar
> > package.
> > &gt; &amp;gt;
> > &gt; &amp;gt;&amp;gt; 在 2020年5月14日，16:20，裴龙武 <
> peilongwu@qq.com <mailto:
> > peilongwu@qq.com&amp;gt;&amp;gt;
> > &gt; 写道：
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; 1. The priority between these tasks
> is also depended
> > on the
> > &gt; dolphin DAG define. When the front task is not finished, it
> not
> > execute
> > &gt; next task.
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; 2. I extend ssh task. I also use
> local params to
> > config ssh host,
> > &gt; user and password.
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; E.g:
> > &gt; &amp;gt;&amp;gt; public static AbstractTask
> > newTask(TaskExecutionContext
> > &gt; taskExecutionContext, Logger logger)
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> throws
> > IllegalArgumentException {
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; Boolean
> enable =
> > &gt;
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; if
> (enable != null
> > &amp;amp;&amp;amp; enable == false ) {
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> > &gt; DummyTask(taskExecutionContext, logger);
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; }
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; switch
> > &gt;
> >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case SHELL:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; ShellTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case
> > PROCEDURE:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; ProcedureTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case SQL:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; SqlTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case MR:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; MapReduceTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case SPARK:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; SparkTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case FLINK:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; FlinkTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case PYTHON:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; PythonTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case HTTP:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; HttpTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case DATAX:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; DataxTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case SQOOP:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; SqoopTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> case SSH:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > return new
> > &gt; SSHTask(taskExecutionContext, logger);
> > &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> default:
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > logger.error("unsupport task
> > &gt; type: {}", taskExecutionContext.getTaskType());
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > throw new
> > &gt; IllegalArgumentException("not support task type");
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; }
> > &gt; &amp;gt;&amp;gt; }
> > &gt; &amp;gt;&amp;gt; 3. I am not sure that it supports
> window or not.
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; ------------------ 原始邮件
> ------------------
> > &gt; &amp;gt;&amp;gt; 发件人: "wenhemin"<whm_777@163.com
> <mailto:
> > whm_777@163.com&amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt; 发送时间: 2020年5月14日(星期四) 下午3:46
> > &gt; &amp;gt;&amp;gt; 收件人: "裴龙武"<peilongwu@qq.com
> <mailto:peilongwu@qq.com
> > &amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt; 主题: Re: [Feature] Support SSH Task
> and Support dummy
> > task like
> > &gt; airflow
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; Sorry, My previous description is
> not very clear.
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt; I want to ask some questions:
> > &gt; &amp;gt;&amp;gt; 1.How to control the priority
> between ssh tasks?
> > There may be
> > &gt; some ssh tasks that have been waiting for execution.
> > &gt; &amp;gt;&amp;gt; 2.I understand what you want to
> solve is the problem
> > of executing
> > &gt; remote ssh scripts in batches.
> > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; So, not
> sure how to use this
> > function.
> > &gt; &amp;gt;&amp;gt; 3.I don't know if this supports
> windows system.
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt; 在 2020年5月13日，20:56，裴龙武 <
> peilongwu@qq.com
> > <mailto:
> > &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt; I use spin lock. Here
> is my code. Of course
> > , it's not
> > &gt; perfect. I just do a test. To my surprise, it is the result
> of the
> > &gt; execution is the same as the AirFlow
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > &gt; 相同的结果。
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt; ------------------ 原始邮件
> ------------------
> > &gt; &amp;gt;&amp;gt;&amp;gt; 发件人: "whm_777"<
> whm_777@163.com <mailto:
> > whm_777@163.com
> > &gt; &amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt; 发送时间: 2020年5月13日(星期三)
> 晚上7:21
> > &gt; &amp;gt;&amp;gt;&amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com <mailto:
> > peilongwu@qq.com&amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt; 主题: Re: [Feature]
> Support SSH Task and
> > Support dummy task
> > &gt; like airflow
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt; You can modify the
> maximum number of linux
> > ssh connections.
> > &gt; &amp;gt;&amp;gt;&amp;gt; If use ssh connection
> pool, How to control
> > the priority of
> > &gt; ssh?
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 在
> 2020年5月13日，18:01，裴龙武 <
> > peilongwu@qq.com <mailto:
> > &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; First 3Q，
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> I&amp;nbsp; use more than 100 task
> > node. But SSH connections
> > &gt; are limited.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> <330BE502@F7F80E73.76C5BB5E.jpg
> > &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> ------------------ 原始邮件
> > ------------------
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 发件人:
> "whm_777"<whm_777@163.com
> > <mailto:whm_777@163.com
> > &gt; &amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 发送时间:
> 2020年5月13日(星期三) 下午5:50
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 收件人: "裴龙武"<
> peilongwu@qq.com <mailto:
> > peilongwu@qq.com
> > &gt; &amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 主题: Re:
> [Feature] Support SSH Task
> > and Support dummy task
> > &gt; like airflow
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; E.g.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> rtn_code=`ssh -o
> > ServerAliveInterval=60 -p xxxx
> > &gt; root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx&amp;gt;
> ‘shell
> > &gt; command&amp;nbsp; &amp;gt;/dev/null
> 2&amp;gt;&amp;amp;1; echo $?'`
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; if [
> "$rtn_code" -eq 0 ]; then
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > echo "成功"
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > exit 0
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; else
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > echo "失败"
> > &gt;
> >
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> > exit 1
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; fi
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Batch shell
> command is not supported.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Multiple
> servers can be split into
> > multiple task nodes.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 在 2020年5月13日，17:40，裴龙武 <
> > peilongwu@qq.com <mailto:
> > &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> Could you give me a
> > example，3Q. 能否给我一个例子，谢谢！
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> By the way, I have more than
> > 100 tasks in one DAG.
> > &gt; These tasks connect two other server to execute. So SSH
> tasks must
> > have
> > &gt; pool to manager. Now I use JSch and realize a simple pool.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 顺带说一下，在我的实际场景中，我有100多个 SSH
> > 任务，这些任务连接两台任务服务器进行任务执行。所以
> > &gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> ------------------ 原始邮件
> > ------------------
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 发件人: "wenhemin"<
> > whm_777@163.com <mailto:
> > &gt; whm_777@163.com&amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 收件人: "dev"<
> > dev@dolphinscheduler.apache.org <mailto:
> > &gt; dev@dolphinscheduler.apache.org&amp;gt;&amp;gt;;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> 主题: Re: [Feature] Support
> > SSH Task and Support dummy
> > &gt; task like airflow
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> The shell node is supports
> > remote calling, and get
> > &gt; the remote command result code.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; 在
> > 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> > &gt; <mailto:peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Dear ALL：
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Support Linux SSH
> > Task 支持 Linux SSH 任务
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> > &gt; 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > &gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; For example, in my
> > project, the workflow's tasks
> > &gt; want to execute shell scripts where are in different server's
> > different
> > &gt; directory. When worker execute these shell scripts, it must
> use the
> > same
> > &gt; user to login these server. Also, the worker can get the
> executing
> > state of
> > &gt; these server. We can config these server 's host,user and
> password.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; SSH Task is very
> > useful for most user SSH
> > &gt; 任务对大多数用户是非常有用的
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; 分布式调度任务所执行的 Shell
> > &gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> > &gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; In
> > dolphinscheduler, the most executing tasks
> > &gt; are in different servers who are not workers. These servers
> also have
> > their
> > &gt; different fixed services. We just have to pass different
> parameters to
> > &gt; schedule these shell scripts to execute.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Python has a module
> > to execute ssh script Python
> > &gt; 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Python
> > 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Python has a module
> > that can execute SSH Shell
> > &gt; script. It's paramiko.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Others 其他内容
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; I found this
> > described in previous feature, but
> > &gt; it was relatively simple.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Feature URL
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; 另外，我通过 Shell Task
> > &gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; In addition, it is
> > very inconvenient for me to
> > &gt; perform remote tasks through Shell Task. Here is my script.
> I don't
> > know if
> > &gt; there's a better way.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; sshpass -p
> > 'password' ssh user@host echo 'ssh
> > &gt; success' echo 'Hello World' -&amp;amp;gt;
> > /home/dolphinscheduler/test/hello.txt
> > &gt; echo 'end'
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Support dummy task
> > like airflow 支持像 Airflow
> > &gt; 中的虚拟任务
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; 场景描述：项目中，有已经产品化的
> > DAG 文件，DAG
> > &gt;
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> > Dummy
> > &gt;
> >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; For example, in my
> > project, it has a productized
> > &gt; DAG file. The file contains different modules, some of which
> are
> > &gt; interdependent and some of which are not. When customers
> purchase
> > different
> > &gt; modules, we need to set some tasks as dummy tasks, which
> some modules
> > are
> > &gt; not purchased and the purchased module is not dependent.
> Because of
> > this
> > &gt; setting, these dummy tasks are actually not executed. The
> benefits of
> > this
> > &gt; setup are product unity and diagram integrity. In airflow,
> these task
> > &gt; execute by dummy operator.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; ** Realize 实现方式**
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Dummy Task
> > 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > &gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; Dummy Task is easy
> > to realize, but it need to
> > &gt; use with other different tasks. When the task's executed
> type is set
> > to
> > &gt; dummy type, the task are executed as a dummy task and the
> real task
> > is not
> > &gt; executed.
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &amp;gt; By the way，I
> > already realize these two&amp;amp;nbsp;
> > &gt; features in my fork branch.&amp;amp;nbsp;Whether the
> follow-up
> > release can be
> > &gt; supported
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> > &gt; &amp;gt;&amp;gt;&amp;gt;
> >
> <SSHClient.java&amp;gt;<SSHPool.java&amp;gt;<SSHTask.java&amp;gt;
> > &gt; &amp;gt;&amp;gt;
> > &gt; &amp;gt;
> > &gt; &amp;gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;gt;
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
Thanks for your advise and suggestions. It may has some exceptions if I implement SSH Task as you said. Now, let me explain my ideas.
e.g
My workflow DAG has 100 or more task, it's all ssh task. I have two questions.
1、We must limit SSH connection and reuse the SSH connections, otherwise the remote server will be refused.
2、If we config all of these task, we have a lot of repeat work to do.
By the way, the way I'm currently verifying is I use local params to config the remote server's host,user,password
------------------ 原始邮件 ------------------
发件人: "lidong dai"<dailidong66@gmail.com>;
发送时间: 2020年5月20日(星期三) 晚上10:56
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
hi, I see you will Jsch, does his license compitable with apache v2
license, I see the following words "
JSch 0.0.* was released under the GNU LGPL license. Later, we have switched
over to a BSD-style license.
" on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, LGPL
license not compitable with apache license. but BSD is ok , so the version
of Jsch is key point
second, Connection Pool is heavy resource, Please use carefully
and I'm a little confused, why there must be add a new Table, other task
types also no seperate table, could you consider task_type and task_json
in t_ds_task_instance ?
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午9:57写道：
> I want to describe my idea.
>
>
> First, we must design a host table that's named t_ds_ssh_host, e.g
>
>
> - id
> - name
> - ip / host
> - user
> - password
> - max_connection
> - create_time
> - update_time
> - user_id
>
>
> Second, Shell can execute in worker server or remote host. We can choose a
> host in host's list.
>
>
> Third, when workflow running, we must maintain the remote host's
> connection pool.
>
>
> Finally, when workflow finish, we must release the connection pool.
>
>
> Why do we have to maintain a connection pool. Because it is very easy to
> get exceptions when it's has many SSH task.
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"lidong dai"<dailidong66@gmail.com&gt;;
> 发送时间:&nbsp;2020年5月20日(星期三) 晚上7:12
> 收件人:&nbsp;"dev"<dev@dolphinscheduler.apache.org&gt;;
>
> 主题:&nbsp;Re: [Feature] Support SSH Task
>
>
>
> glad to hear that you will implement this feature
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <peilongwu@qq.com&gt; 于2020年5月20日周三 下午3:47写道：
>
> &gt; My code is not perfect yet. I will write a detailed design document.
> Then
> &gt; I will realize this feature about our discussion result.
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:&amp;nbsp;"wenhemin"<whm_777@163.com&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2020年5月18日(星期一) 晚上7:50
> &gt; 收件人:&amp;nbsp;"裴龙武"<peilongwu@qq.com&amp;gt;;"dev"<
> dev@dolphinscheduler.apache.org
> &gt; &amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> &gt;
> &gt;
> &gt;
> &gt; Thanks for writing detailed documentation. I think this is also a
> missing
> &gt; feature of DS.
> &gt; About the extension point:
> &gt; 1.Can ssh tasks be merged into shell tasks. Essentially, they all
> execute
> &gt; shell commands.
> &gt; 2.About dummy task, DS has the function of disable nodes, I do n’t
> know if
> &gt; this requirement is met.
> &gt;
> &gt; The script from AirFlow to Dolphin is great.
> &gt;
> &gt; &amp;gt; 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com&amp;gt; 写道：
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; OK, 3Q!
> &gt; &amp;gt;
> &gt; &amp;gt; First, I will ensure that open source can use.
> &gt; &amp;gt;
> &gt; &amp;gt; Second, I think we must discuss deeply. I write a more
> detailed
> &gt; document. You can check the attachment. I also send the document to
> &gt; DaiLidong.
> &gt; &amp;gt;
> &gt; &amp;gt; Third,&amp;nbsp; I'll give you the error of not using SSH
> connection pool.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; ------------------ 原始邮件 ------------------
> &gt; &amp;gt; 发件人: "wenhemin"<whm_777@163.com&amp;gt;;
> &gt; &amp;gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> &gt; &amp;gt; 收件人: "裴龙武"<peilongwu@qq.com&amp;gt;;
> &gt; &amp;gt; 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> &gt; &amp;gt;
> &gt; &amp;gt; Great!
> &gt; &amp;gt; I think, Can ssh tasks be merged into shell tasks,&amp;nbsp;
> execute
> &gt; script locally or remotely, Configure on the front end.
> &gt; &amp;gt; About ssh connect pool, I did not find it necessary to use
> the
> &gt; connection pool.
> &gt; &amp;gt;
> &gt; &amp;gt; BTW, Look at the code to introduce additional jar packages,
> You also
> &gt; need to ensure that open source can use the license of this jar
> package.
> &gt; &amp;gt;
> &gt; &amp;gt;&amp;gt; 在 2020年5月14日，16:20，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com&amp;gt;&amp;gt;
> &gt; 写道：
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; 1. The priority between these tasks is also depended
> on the
> &gt; dolphin DAG define. When the front task is not finished, it not
> execute
> &gt; next task.
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; 2. I extend ssh task. I also use local params to
> config ssh host,
> &gt; user and password.
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; E.g:
> &gt; &amp;gt;&amp;gt; public static AbstractTask
> newTask(TaskExecutionContext
> &gt; taskExecutionContext, Logger logger)
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; throws
> IllegalArgumentException {
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; Boolean enable =
> &gt;
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; if (enable != null
> &amp;amp;&amp;amp; enable == false ) {
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; return new
> &gt; DummyTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; }
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; switch
> &gt;
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SHELL:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; ShellTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case
> PROCEDURE:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; ProcedureTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SQL:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; SqlTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case MR:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; MapReduceTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SPARK:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; SparkTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case FLINK:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; FlinkTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case PYTHON:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; PythonTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case HTTP:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; HttpTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case DATAX:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; DataxTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SQOOP:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; SqoopTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SSH:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> return new
> &gt; SSHTask(taskExecutionContext, logger);
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; default:
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> logger.error("unsupport task
> &gt; type: {}", taskExecutionContext.getTaskType());
> &gt;
> &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> throw new
> &gt; IllegalArgumentException("not support task type");
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; }
> &gt; &amp;gt;&amp;gt; }
> &gt; &amp;gt;&amp;gt; 3. I am not sure that it supports window or not.
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; ------------------ 原始邮件 ------------------
> &gt; &amp;gt;&amp;gt; 发件人: "wenhemin"<whm_777@163.com <mailto:
> whm_777@163.com&amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt; 发送时间: 2020年5月14日(星期四) 下午3:46
> &gt; &amp;gt;&amp;gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com
> &amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt; 主题: Re: [Feature] Support SSH Task and Support dummy
> task like
> &gt; airflow
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; Sorry, My previous description is not very clear.
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt; I want to ask some questions:
> &gt; &amp;gt;&amp;gt; 1.How to control the priority between ssh tasks?
> There may be
> &gt; some ssh tasks that have been waiting for execution.
> &gt; &amp;gt;&amp;gt; 2.I understand what you want to solve is the problem
> of executing
> &gt; remote ssh scripts in batches.
> &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; So, not sure how to use this
> function.
> &gt; &amp;gt;&amp;gt; 3.I don't know if this supports windows system.
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt; 在 2020年5月13日，20:56，裴龙武 <peilongwu@qq.com
> <mailto:
> &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt; I use spin lock. Here is my code. Of course
> , it's not
> &gt; perfect. I just do a test. To my surprise, it is the result of the
> &gt; execution is the same as the AirFlow
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> &gt; 相同的结果。
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt; ------------------ 原始邮件 ------------------
> &gt; &amp;gt;&amp;gt;&amp;gt; 发件人: "whm_777"<whm_777@163.com <mailto:
> whm_777@163.com
> &gt; &amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt; 发送时间: 2020年5月13日(星期三) 晚上7:21
> &gt; &amp;gt;&amp;gt;&amp;gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:
> peilongwu@qq.com&amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt; 主题: Re: [Feature] Support SSH Task and
> Support dummy task
> &gt; like airflow
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt; You can modify the maximum number of linux
> ssh connections.
> &gt; &amp;gt;&amp;gt;&amp;gt; If use ssh connection pool, How to control
> the priority of
> &gt; ssh?
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 在 2020年5月13日，18:01，裴龙武 <
> peilongwu@qq.com <mailto:
> &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; First 3Q，
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; I&amp;nbsp; use more than 100 task
> node. But SSH connections
> &gt; are limited.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt;
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; <330BE502@F7F80E73.76C5BB5E.jpg
> &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ------------------ 原始邮件
> ------------------
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 发件人: "whm_777"<whm_777@163.com
> <mailto:whm_777@163.com
> &gt; &amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:50
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:
> peilongwu@qq.com
> &gt; &amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; 主题: Re: [Feature] Support SSH Task
> and Support dummy task
> &gt; like airflow
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; E.g.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; rtn_code=`ssh -o
> ServerAliveInterval=60 -p xxxx
> &gt; root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx&amp;gt; ‘shell
> &gt; command&amp;nbsp; &amp;gt;/dev/null 2&amp;gt;&amp;amp;1; echo $?'`
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; if [ "$rtn_code" -eq 0 ]; then
> &gt;
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> echo "成功"
> &gt;
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> exit 0
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; else
> &gt;
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> echo "失败"
> &gt;
> &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> exit 1
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; fi
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Batch shell command is not supported.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Multiple servers can be split into
> multiple task nodes.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 在 2020年5月13日，17:40，裴龙武 <
> peilongwu@qq.com <mailto:
> &gt; peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; Could you give me a
> example，3Q. 能否给我一个例子，谢谢！
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; By the way, I have more than
> 100 tasks in one DAG.
> &gt; These tasks connect two other server to execute. So SSH tasks must
> have
> &gt; pool to manager. Now I use JSch and realize a simple pool.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> 任务，这些任务连接两台任务服务器进行任务执行。所以
> &gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ------------------ 原始邮件
> ------------------
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 发件人: "wenhemin"<
> whm_777@163.com <mailto:
> &gt; whm_777@163.com&amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 收件人: "dev"<
> dev@dolphinscheduler.apache.org <mailto:
> &gt; dev@dolphinscheduler.apache.org&amp;gt;&amp;gt;;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; 主题: Re: [Feature] Support
> SSH Task and Support dummy
> &gt; task like airflow
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; The shell node is supports
> remote calling, and get
> &gt; the remote command result code.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; 在
> 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> &gt; <mailto:peilongwu@qq.com&amp;gt;&amp;gt; 写道：
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dear ALL：
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Support Linux SSH
> Task 支持 Linux SSH 任务
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> &gt; 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> &gt; 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; For example, in my
> project, the workflow's tasks
> &gt; want to execute shell scripts where are in different server's
> different
> &gt; directory. When worker execute these shell scripts, it must use the
> same
> &gt; user to login these server. Also, the worker can get the executing
> state of
> &gt; these server. We can config these server 's host,user and password.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; SSH Task is very
> useful for most user SSH
> &gt; 任务对大多数用户是非常有用的
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; 分布式调度任务所执行的 Shell
> &gt; 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> &gt; 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; In
> dolphinscheduler, the most executing tasks
> &gt; are in different servers who are not workers. These servers also have
> their
> &gt; different fixed services. We just have to pass different parameters to
> &gt; schedule these shell scripts to execute.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python has a module
> to execute ssh script Python
> &gt; 有固定的工具包，可执行这些SSH Shell 脚本
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python
> 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python has a module
> that can execute SSH Shell
> &gt; script. It's paramiko.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Others 其他内容
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; I found this
> described in previous feature, but
> &gt; it was relatively simple.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Feature URL
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; 另外，我通过 Shell Task
> &gt; 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; In addition, it is
> very inconvenient for me to
> &gt; perform remote tasks through Shell Task. Here is my script. I don't
> know if
> &gt; there's a better way.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; sshpass -p
> 'password' ssh user@host echo 'ssh
> &gt; success' echo 'Hello World' -&amp;amp;gt;
> /home/dolphinscheduler/test/hello.txt
> &gt; echo 'end'
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Support dummy task
> like airflow 支持像 Airflow
> &gt; 中的虚拟任务
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; 场景描述：项目中，有已经产品化的
> DAG 文件，DAG
> &gt; 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> Dummy
> &gt;
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; For example, in my
> project, it has a productized
> &gt; DAG file. The file contains different modules, some of which are
> &gt; interdependent and some of which are not. When customers purchase
> different
> &gt; modules, we need to set some tasks as dummy tasks, which some modules
> are
> &gt; not purchased and the purchased module is not dependent. Because of
> this
> &gt; setting, these dummy tasks are actually not executed. The benefits of
> this
> &gt; setup are product unity and diagram integrity. In airflow, these task
> &gt; execute by dummy operator.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ** Realize 实现方式**
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dummy Task
> 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> &gt; dummy 时，实际的任务不执行，执行 Dummy Task。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dummy Task is easy
> to realize, but it need to
> &gt; use with other different tasks. When the task's executed type is set
> to
> &gt; dummy type, the task are executed as a dummy task and the real task
> is not
> &gt; executed.
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; By the way，I
> already realize these two&amp;amp;nbsp;
> &gt; features in my fork branch.&amp;amp;nbsp;Whether the follow-up
> release can be
> &gt; supported
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> &gt; &amp;gt;&amp;gt;&amp;gt;
> <SSHClient.java&amp;gt;<SSHPool.java&amp;gt;<SSHTask.java&amp;gt;
> &gt; &amp;gt;&amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;gt;
Re: [Feature] Support SSH Task
Posted by lidong dai <da...@gmail.com>.
hi, I see you will Jsch, does his license compitable with apache v2
license, I see the following words "
JSch 0.0.* was released under the GNU LGPL license. Later, we have switched
over to a BSD-style license.
" on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, LGPL
license not compitable with apache license. but BSD is ok , so the version
of Jsch is key point
second, Connection Pool is heavy resource, Please use carefully
and I'm a little confused, why there must be add a new Table, other task
types also no seperate table, could you consider task_type and task_json
in t_ds_task_instance ?
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <pe...@qq.com> 于2020年5月20日周三 下午9:57写道：
> I want to describe my idea.
>
>
> First, we must design a host table that's named t_ds_ssh_host, e.g
>
>
> - id
> - name
> - ip / host
> - user
> - password
> - max_connection
> - create_time
> - update_time
> - user_id
>
>
> Second, Shell can execute in worker server or remote host. We can choose a
> host in host's list.
>
>
> Third, when workflow running, we must maintain the remote host's
> connection pool.
>
>
> Finally, when workflow finish, we must release the connection pool.
>
>
> Why do we have to maintain a connection pool. Because it is very easy to
> get exceptions when it's has many SSH task.
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "lidong dai"<dailidong66@gmail.com>;
> 发送时间: 2020年5月20日(星期三) 晚上7:12
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Feature] Support SSH Task
>
>
>
> glad to hear that you will implement this feature
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> 裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午3:47写道：
>
> > My code is not perfect yet. I will write a detailed design document.
> Then
> > I will realize this feature about our discussion result.
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:&nbsp;"wenhemin"<whm_777@163.com&gt;;
> > 发送时间:&nbsp;2020年5月18日(星期一) 晚上7:50
> > 收件人:&nbsp;"裴龙武"<peilongwu@qq.com&gt;;"dev"<
> dev@dolphinscheduler.apache.org
> > &gt;;
> >
> > 主题:&nbsp;Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> >
> >
> >
> > Thanks for writing detailed documentation. I think this is also a
> missing
> > feature of DS.
> > About the extension point:
> > 1.Can ssh tasks be merged into shell tasks. Essentially, they all
> execute
> > shell commands.
> > 2.About dummy task, DS has the function of disable nodes, I do n’t
> know if
> > this requirement is met.
> >
> > The script from AirFlow to Dolphin is great.
> >
> > &gt; 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com&gt; 写道：
> > &gt;
> > &gt;
> > &gt; OK, 3Q!
> > &gt;
> > &gt; First, I will ensure that open source can use.
> > &gt;
> > &gt; Second, I think we must discuss deeply. I write a more
> detailed
> > document. You can check the attachment. I also send the document to
> > DaiLidong.
> > &gt;
> > &gt; Third,&nbsp; I'll give you the error of not using SSH
> connection pool.
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt; ------------------ 原始邮件 ------------------
> > &gt; 发件人: "wenhemin"<whm_777@163.com&gt;;
> > &gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> > &gt; 收件人: "裴龙武"<peilongwu@qq.com&gt;;
> > &gt; 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> > &gt;
> > &gt; Great!
> > &gt; I think, Can ssh tasks be merged into shell tasks,&nbsp;
> execute
> > script locally or remotely, Configure on the front end.
> > &gt; About ssh connect pool, I did not find it necessary to use
> the
> > connection pool.
> > &gt;
> > &gt; BTW, Look at the code to introduce additional jar packages,
> You also
> > need to ensure that open source can use the license of this jar
> package.
> > &gt;
> > &gt;&gt; 在 2020年5月14日，16:20，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com&gt;&gt;
> > 写道：
> > &gt;&gt;
> > &gt;&gt;
> > &gt;&gt; 1. The priority between these tasks is also depended
> on the
> > dolphin DAG define. When the front task is not finished, it not
> execute
> > next task.
> > &gt;&gt;
> > &gt;&gt; 2. I extend ssh task. I also use local params to
> config ssh host,
> > user and password.
> > &gt;&gt;
> > &gt;&gt; E.g:
> > &gt;&gt; public static AbstractTask
> newTask(TaskExecutionContext
> > taskExecutionContext, Logger logger)
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; throws
> IllegalArgumentException {
> > &gt;&gt;&nbsp;&nbsp; Boolean enable =
> >
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> > &gt;&gt;&nbsp;&nbsp; if (enable != null
> &amp;&amp; enable == false ) {
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; return new
> > DummyTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp; }
> > &gt;&gt;&nbsp;&nbsp; switch
> >
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SHELL:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > ShellTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case
> PROCEDURE:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > ProcedureTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SQL:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > SqlTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case MR:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > MapReduceTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SPARK:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > SparkTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case FLINK:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > FlinkTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case PYTHON:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > PythonTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case HTTP:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > HttpTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case DATAX:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > DataxTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SQOOP:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > SqoopTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SSH:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> return new
> > SSHTask(taskExecutionContext, logger);
> > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; default:
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> logger.error("unsupport task
> > type: {}", taskExecutionContext.getTaskType());
> >
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> throw new
> > IllegalArgumentException("not support task type");
> > &gt;&gt;&nbsp;&nbsp; }
> > &gt;&gt; }
> > &gt;&gt; 3. I am not sure that it supports window or not.
> > &gt;&gt;
> > &gt;&gt;
> > &gt;&gt;
> > &gt;&gt; ------------------ 原始邮件 ------------------
> > &gt;&gt; 发件人: "wenhemin"<whm_777@163.com <mailto:
> whm_777@163.com&gt;&gt;;
> > &gt;&gt; 发送时间: 2020年5月14日(星期四) 下午3:46
> > &gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com
> &gt;&gt;;
> > &gt;&gt; 主题: Re: [Feature] Support SSH Task and Support dummy
> task like
> > airflow
> > &gt;&gt;
> > &gt;&gt; Sorry, My previous description is not very clear.
> > &gt;&gt;
> > &gt;&gt; I want to ask some questions:
> > &gt;&gt; 1.How to control the priority between ssh tasks?
> There may be
> > some ssh tasks that have been waiting for execution.
> > &gt;&gt; 2.I understand what you want to solve is the problem
> of executing
> > remote ssh scripts in batches.
> > &gt;&gt;&nbsp;&nbsp; So, not sure how to use this
> function.
> > &gt;&gt; 3.I don't know if this supports windows system.
> > &gt;&gt;
> > &gt;&gt;&gt; 在 2020年5月13日，20:56，裴龙武 <peilongwu@qq.com
> <mailto:
> > peilongwu@qq.com&gt;&gt; 写道：
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> > &gt;&gt;&gt; I use spin lock. Here is my code. Of course
> , it's not
> > perfect. I just do a test. To my surprise, it is the result of the
> > execution is the same as the AirFlow
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> > 相同的结果。
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> > &gt;&gt;&gt; ------------------ 原始邮件 ------------------
> > &gt;&gt;&gt; 发件人: "whm_777"<whm_777@163.com <mailto:
> whm_777@163.com
> > &gt;&gt;;
> > &gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 晚上7:21
> > &gt;&gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:
> peilongwu@qq.com&gt;&gt;;
> > &gt;&gt;&gt; 主题: Re: [Feature] Support SSH Task and
> Support dummy task
> > like airflow
> > &gt;&gt;&gt;
> > &gt;&gt;&gt; You can modify the maximum number of linux
> ssh connections.
> > &gt;&gt;&gt; If use ssh connection pool, How to control
> the priority of
> > ssh?
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; 在 2020年5月13日，18:01，裴龙武 <
> peilongwu@qq.com <mailto:
> > peilongwu@qq.com&gt;&gt; 写道：
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; First 3Q，
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; I&nbsp; use more than 100 task
> node. But SSH connections
> > are limited.
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;
> >
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> > &gt;&gt;&gt;&gt; <330BE502@F7F80E73.76C5BB5E.jpg
> &gt;
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; ------------------ 原始邮件
> ------------------
> > &gt;&gt;&gt;&gt; 发件人: "whm_777"<whm_777@163.com
> <mailto:whm_777@163.com
> > &gt;&gt;;
> > &gt;&gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 下午5:50
> > &gt;&gt;&gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:
> peilongwu@qq.com
> > &gt;&gt;;
> > &gt;&gt;&gt;&gt; 主题: Re: [Feature] Support SSH Task
> and Support dummy task
> > like airflow
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; E.g.
> > &gt;&gt;&gt;&gt; rtn_code=`ssh -o
> ServerAliveInterval=60 -p xxxx
> > root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx&gt; ‘shell
> > command&nbsp; &gt;/dev/null 2&gt;&amp;1; echo $?'`
> > &gt;&gt;&gt;&gt; if [ "$rtn_code" -eq 0 ]; then
> >
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> echo "成功"
> >
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> exit 0
> > &gt;&gt;&gt;&gt; else
> >
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> echo "失败"
> >
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> exit 1
> > &gt;&gt;&gt;&gt; fi
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt; Batch shell command is not supported.
> > &gt;&gt;&gt;&gt; Multiple servers can be split into
> multiple task nodes.
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; 在 2020年5月13日，17:40，裴龙武 <
> peilongwu@qq.com <mailto:
> > peilongwu@qq.com&gt;&gt; 写道：
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; Could you give me a
> example，3Q. 能否给我一个例子，谢谢！
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; By the way, I have more than
> 100 tasks in one DAG.
> > These tasks connect two other server to execute. So SSH tasks must
> have
> > pool to manager. Now I use JSch and realize a simple pool.
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; 顺带说一下，在我的实际场景中，我有100多个 SSH
> 任务，这些任务连接两台任务服务器进行任务执行。所以
> > SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; ------------------ 原始邮件
> ------------------
> > &gt;&gt;&gt;&gt;&gt; 发件人: "wenhemin"<
> whm_777@163.com <mailto:
> > whm_777@163.com&gt;&gt;;
> > &gt;&gt;&gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> > &gt;&gt;&gt;&gt;&gt; 收件人: "dev"<
> dev@dolphinscheduler.apache.org <mailto:
> > dev@dolphinscheduler.apache.org&gt;&gt;;
> > &gt;&gt;&gt;&gt;&gt; 主题: Re: [Feature] Support
> SSH Task and Support dummy
> > task like airflow
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; The shell node is supports
> remote calling, and get
> > the remote command result code.
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; 在
> 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> > <mailto:peilongwu@qq.com&gt;&gt; 写道：
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Dear ALL：
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Support Linux SSH
> Task 支持 Linux SSH 任务
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> > 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; For example, in my
> project, the workflow's tasks
> > want to execute shell scripts where are in different server's
> different
> > directory. When worker execute these shell scripts, it must use the
> same
> > user to login these server. Also, the worker can get the executing
> state of
> > these server. We can config these server 's host,user and password.
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; SSH Task is very
> useful for most user SSH
> > 任务对大多数用户是非常有用的
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; 分布式调度任务所执行的 Shell
> > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; In
> dolphinscheduler, the most executing tasks
> > are in different servers who are not workers. These servers also have
> their
> > different fixed services. We just have to pass different parameters to
> > schedule these shell scripts to execute.
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Python has a module
> to execute ssh script Python
> > 有固定的工具包，可执行这些SSH Shell 脚本
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Python
> 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Python has a module
> that can execute SSH Shell
> > script. It's paramiko.
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Others 其他内容
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; I found this
> described in previous feature, but
> > it was relatively simple.
> > &gt;&gt;&gt;&gt;&gt; &gt; Feature URL
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; 另外，我通过 Shell Task
> > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; In addition, it is
> very inconvenient for me to
> > perform remote tasks through Shell Task. Here is my script. I don't
> know if
> > there's a better way.
> > &gt;&gt;&gt;&gt;&gt; &gt; sshpass -p
> 'password' ssh user@host echo 'ssh
> > success' echo 'Hello World' -&amp;gt;
> /home/dolphinscheduler/test/hello.txt
> > echo 'end'
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Support dummy task
> like airflow 支持像 Airflow
> > 中的虚拟任务
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; 场景描述：项目中，有已经产品化的
> DAG 文件，DAG
> > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为
> Dummy
> >
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; For example, in my
> project, it has a productized
> > DAG file. The file contains different modules, some of which are
> > interdependent and some of which are not. When customers purchase
> different
> > modules, we need to set some tasks as dummy tasks, which some modules
> are
> > not purchased and the purchased module is not dependent. Because of
> this
> > setting, these dummy tasks are actually not executed. The benefits of
> this
> > setup are product unity and diagram integrity. In airflow, these task
> > execute by dummy operator.
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; ** Realize 实现方式**
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Dummy Task
> 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> > dummy 时，实际的任务不执行，执行 Dummy Task。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; Dummy Task is easy
> to realize, but it need to
> > use with other different tasks. When the task's executed type is set
> to
> > dummy type, the task are executed as a dummy task and the real task
> is not
> > executed.
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> > &gt;&gt;&gt;&gt;&gt; &gt;
> > &gt;&gt;&gt;&gt;&gt; &gt; By the way，I
> already realize these two&amp;nbsp;
> > features in my fork branch.&amp;nbsp;Whether the follow-up
> release can be
> > supported
> > &gt;&gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;&gt;
> > &gt;&gt;&gt;
> > &gt;&gt;&gt;
> <SSHClient.java&gt;<SSHPool.java&gt;<SSHTask.java&gt;
> > &gt;&gt;
> > &gt;
> > &gt; <项目场景中关于Dolphin的一些扩展点.pdf&gt;
回复： [Feature] Support SSH Task
Posted by 裴龙武 <pe...@qq.com>.
I want to describe my idea.
First, we must design a host table that's named t_ds_ssh_host, e.g
- id
- name
- ip / host
- user
- password
- max_connection
- create_time
- update_time
- user_id
Second, Shell can execute in worker server or remote host. We can choose a host in host's list.
Third, when workflow running, we must maintain the remote host's connection pool.
Finally, when workflow finish, we must release the connection pool.
Why do we have to maintain a connection pool. Because it is very easy to get exceptions when it's has many SSH task.
------------------ 原始邮件 ------------------
发件人: "lidong dai"<dailidong66@gmail.com>;
发送时间: 2020年5月20日(星期三) 晚上7:12
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Feature] Support SSH Task
glad to hear that you will implement this feature
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
裴龙武 <peilongwu@qq.com> 于2020年5月20日周三 下午3:47写道：
> My code is not perfect yet. I will write a detailed design document. Then
> I will realize this feature about our discussion result.
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"wenhemin"<whm_777@163.com&gt;;
> 发送时间:&nbsp;2020年5月18日(星期一) 晚上7:50
> 收件人:&nbsp;"裴龙武"<peilongwu@qq.com&gt;;"dev"<dev@dolphinscheduler.apache.org
> &gt;;
>
> 主题:&nbsp;Re: [Feature] Support SSH Task and Support dummy task like airflow
>
>
>
> Thanks for writing detailed documentation. I think this is also a missing
> feature of DS.
> About the extension point:
> 1.Can ssh tasks be merged into shell tasks. Essentially, they all execute
> shell commands.
> 2.About dummy task, DS has the function of disable nodes, I do n’t know if
> this requirement is met.
>
> The script from AirFlow to Dolphin is great.
>
> &gt; 在 2020年5月18日，09:28，裴龙武 <peilongwu@qq.com&gt; 写道：
> &gt;
> &gt;
> &gt; OK, 3Q!
> &gt;
> &gt; First, I will ensure that open source can use.
> &gt;
> &gt; Second, I think we must discuss deeply. I write a more detailed
> document. You can check the attachment. I also send the document to
> DaiLidong.
> &gt;
> &gt; Third,&nbsp; I'll give you the error of not using SSH connection pool.
> &gt;
> &gt;
> &gt;
> &gt;
> &gt; ------------------ 原始邮件 ------------------
> &gt; 发件人: "wenhemin"<whm_777@163.com&gt;;
> &gt; 发送时间: 2020年5月14日(星期四) 晚上7:26
> &gt; 收件人: "裴龙武"<peilongwu@qq.com&gt;;
> &gt; 主题: Re: [Feature] Support SSH Task and Support dummy task like airflow
> &gt;
> &gt; Great!
> &gt; I think, Can ssh tasks be merged into shell tasks,&nbsp; execute
> script locally or remotely, Configure on the front end.
> &gt; About ssh connect pool, I did not find it necessary to use the
> connection pool.
> &gt;
> &gt; BTW, Look at the code to introduce additional jar packages, You also
> need to ensure that open source can use the license of this jar package.
> &gt;
> &gt;&gt; 在 2020年5月14日，16:20，裴龙武 <peilongwu@qq.com <mailto:peilongwu@qq.com&gt;&gt;
> 写道：
> &gt;&gt;
> &gt;&gt;
> &gt;&gt; 1. The priority between these tasks is also depended on the
> dolphin DAG define. When the front task is not finished, it not execute
> next task.
> &gt;&gt;
> &gt;&gt; 2. I extend ssh task. I also use local params to config ssh host,
> user and password.
> &gt;&gt;
> &gt;&gt; E.g:
> &gt;&gt; public static AbstractTask newTask(TaskExecutionContext
> taskExecutionContext, Logger logger)
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; throws IllegalArgumentException {
> &gt;&gt;&nbsp;&nbsp; Boolean enable =
> JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable");
> &gt;&gt;&nbsp;&nbsp; if (enable != null &amp;&amp; enable == false ) {
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; return new
> DummyTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp; }
> &gt;&gt;&nbsp;&nbsp; switch
> (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) {
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SHELL:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> ShellTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case PROCEDURE:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> ProcedureTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SQL:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> SqlTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case MR:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> MapReduceTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SPARK:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> SparkTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case FLINK:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> FlinkTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case PYTHON:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> PythonTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case HTTP:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> HttpTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case DATAX:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> DataxTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SQOOP:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> SqoopTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; case SSH:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return new
> SSHTask(taskExecutionContext, logger);
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; default:
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; logger.error("unsupport task
> type: {}", taskExecutionContext.getTaskType());
> &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; throw new
> IllegalArgumentException("not support task type");
> &gt;&gt;&nbsp;&nbsp; }
> &gt;&gt; }
> &gt;&gt; 3. I am not sure that it supports window or not.
> &gt;&gt;
> &gt;&gt;
> &gt;&gt;
> &gt;&gt; ------------------ 原始邮件 ------------------
> &gt;&gt; 发件人: "wenhemin"<whm_777@163.com <mailto:whm_777@163.com&gt;&gt;;
> &gt;&gt; 发送时间: 2020年5月14日(星期四) 下午3:46
> &gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com&gt;&gt;;
> &gt;&gt; 主题: Re: [Feature] Support SSH Task and Support dummy task like
> airflow
> &gt;&gt;
> &gt;&gt; Sorry, My previous description is not very clear.
> &gt;&gt;
> &gt;&gt; I want to ask some questions:
> &gt;&gt; 1.How to control the priority between ssh tasks? There may be
> some ssh tasks that have been waiting for execution.
> &gt;&gt; 2.I understand what you want to solve is the problem of executing
> remote ssh scripts in batches.
> &gt;&gt;&nbsp;&nbsp; So, not sure how to use this function.
> &gt;&gt; 3.I don't know if this supports windows system.
> &gt;&gt;
> &gt;&gt;&gt; 在 2020年5月13日，20:56，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com&gt;&gt; 写道：
> &gt;&gt;&gt;
> &gt;&gt;&gt;
> &gt;&gt;&gt; I use spin lock. Here is my code. Of course , it's not
> perfect. I just do a test. To my surprise, it is the result of the
> execution is the same as the AirFlow
> &gt;&gt;&gt;
> &gt;&gt;&gt; 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow
> 相同的结果。
> &gt;&gt;&gt;
> &gt;&gt;&gt;
> &gt;&gt;&gt;
> &gt;&gt;&gt;
> &gt;&gt;&gt; ------------------ 原始邮件 ------------------
> &gt;&gt;&gt; 发件人: "whm_777"<whm_777@163.com <mailto:whm_777@163.com
> &gt;&gt;;
> &gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 晚上7:21
> &gt;&gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com&gt;&gt;;
> &gt;&gt;&gt; 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> &gt;&gt;&gt;
> &gt;&gt;&gt; You can modify the maximum number of linux ssh connections.
> &gt;&gt;&gt; If use ssh connection pool, How to control the priority of
> ssh?
> &gt;&gt;&gt;
> &gt;&gt;&gt;&gt; 在 2020年5月13日，18:01，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com&gt;&gt; 写道：
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt; First 3Q，
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt; I&nbsp; use more than 100 task node. But SSH connections
> are limited.
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;
> 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。
> &gt;&gt;&gt;&gt; <330BE502@F7F80E73.76C5BB5E.jpg&gt;
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt; ------------------ 原始邮件 ------------------
> &gt;&gt;&gt;&gt; 发件人: "whm_777"<whm_777@163.com <mailto:whm_777@163.com
> &gt;&gt;;
> &gt;&gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 下午5:50
> &gt;&gt;&gt;&gt; 收件人: "裴龙武"<peilongwu@qq.com <mailto:peilongwu@qq.com
> &gt;&gt;;
> &gt;&gt;&gt;&gt; 主题: Re: [Feature] Support SSH Task and Support dummy task
> like airflow
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt; E.g.
> &gt;&gt;&gt;&gt; rtn_code=`ssh -o ServerAliveInterval=60 -p xxxx
> root@xxx.xxx.xxx.xxx <mailto:root@xxx.xxx.xxx.xxx&gt; ‘shell
> command&nbsp; &gt;/dev/null 2&gt;&amp;1; echo $?'`
> &gt;&gt;&gt;&gt; if [ "$rtn_code" -eq 0 ]; then
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; echo "成功"
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; exit 0
> &gt;&gt;&gt;&gt; else
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; echo "失败"
> &gt;&gt;&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; exit 1
> &gt;&gt;&gt;&gt; fi
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt; Batch shell command is not supported.
> &gt;&gt;&gt;&gt; Multiple servers can be split into multiple task nodes.
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; 在 2020年5月13日，17:40，裴龙武 <peilongwu@qq.com <mailto:
> peilongwu@qq.com&gt;&gt; 写道：
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; Could you give me a example，3Q. 能否给我一个例子，谢谢！
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; By the way, I have more than 100 tasks in one DAG.
> These tasks connect two other server to execute. So SSH tasks must have
> pool to manager. Now I use JSch and realize a simple pool.
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; 顺带说一下，在我的实际场景中，我有100多个 SSH 任务，这些任务连接两台任务服务器进行任务执行。所以
> SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 JSch，并实现了一个简单的连接池。
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; ------------------ 原始邮件 ------------------
> &gt;&gt;&gt;&gt;&gt; 发件人: "wenhemin"<whm_777@163.com <mailto:
> whm_777@163.com&gt;&gt;;
> &gt;&gt;&gt;&gt;&gt; 发送时间: 2020年5月13日(星期三) 下午5:24
> &gt;&gt;&gt;&gt;&gt; 收件人: "dev"<dev@dolphinscheduler.apache.org <mailto:
> dev@dolphinscheduler.apache.org&gt;&gt;;
> &gt;&gt;&gt;&gt;&gt; 主题: Re: [Feature] Support SSH Task and Support dummy
> task like airflow
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; The shell node is supports remote calling, and get
> the remote command result code.
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 在 2020年5月13日，15:16，裴龙武 <peilongwu@qq.com
> <mailto:peilongwu@qq.com&gt;&gt; 写道：
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Dear ALL：
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Support Linux SSH Task 支持 Linux SSH 任务
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell
> 脚本是保存在业务服务器的固定目录。当 Worker 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell
> 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; For example, in my project, the workflow's tasks
> want to execute shell scripts where are in different server's different
> directory. When worker execute these shell scripts, it must use the same
> user to login these server. Also, the worker can get the executing state of
> these server. We can config these server 's host,user and password.
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; SSH Task is very useful for most user SSH
> 任务对大多数用户是非常有用的
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 分布式调度任务所执行的 Shell
> 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 Worker
> 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; In dolphinscheduler, the most executing tasks
> are in different servers who are not workers. These servers also have their
> different fixed services. We just have to pass different parameters to
> schedule these shell scripts to execute.
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Python has a module to execute ssh script Python
> 有固定的工具包，可执行这些SSH Shell 脚本
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Python 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Python has a module that can execute SSH Shell
> script. It's paramiko.
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Others 其他内容
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; I found this described in previous feature, but
> it was relatively simple.
> &gt;&gt;&gt;&gt;&gt; &gt; Feature URL
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 另外，我通过 Shell Task
> 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; In addition, it is very inconvenient for me to
> perform remote tasks through Shell Task. Here is my script. I don't know if
> there's a better way.
> &gt;&gt;&gt;&gt;&gt; &gt; sshpass -p 'password' ssh user@host echo 'ssh
> success' echo 'Hello World' -&amp;gt; /home/dolphinscheduler/test/hello.txt
> echo 'end'
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Support dummy task like airflow 支持像 Airflow
> 中的虚拟任务
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; 场景描述：项目中，有已经产品化的 DAG 文件，DAG
> 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为 Dummy
> Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; For example, in my project, it has a productized
> DAG file. The file contains different modules, some of which are
> interdependent and some of which are not. When customers purchase different
> modules, we need to set some tasks as dummy tasks, which some modules are
> not purchased and the purchased module is not dependent. Because of this
> setting, these dummy tasks are actually not executed. The benefits of this
> setup are product unity and diagram integrity. In airflow, these task
> execute by dummy operator.
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; ** Realize 实现方式**
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Dummy Task 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为
> dummy 时，实际的任务不执行，执行 Dummy Task。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; Dummy Task is easy to realize, but it need to
> use with other different tasks. When the task's executed type is set to
> dummy type, the task are executed as a dummy task and the real task is not
> executed.
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt;
> 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。
> &gt;&gt;&gt;&gt;&gt; &gt;
> &gt;&gt;&gt;&gt;&gt; &gt; By the way，I already realize these two&amp;nbsp;
> features in my fork branch.&amp;nbsp;Whether the follow-up release can be
> supported
> &gt;&gt;&gt;&gt;&gt;
> &gt;&gt;&gt;&gt;
> &gt;&gt;&gt;
> &gt;&gt;&gt; <SSHClient.java&gt;<SSHPool.java&gt;<SSHTask.java&gt;
> &gt;&gt;
> &gt;
> &gt; <项目场景中关于Dolphin的一些扩展点.pdf&gt;