You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by 张世鸣 <36...@qq.com> on 2020/03/11 11:18:43 UTC
add custom configuration Feature for datax 增加datax自定义配置功能
https://github.com/apache/incubator-dolphinscheduler/issues/2102
Although the datax plugin has not been officially released, I had read the relevant source code and found that it only supports a few reading and writing methods in the figure, which cannot cover my current needs. if it is possible to add the function of a custom datax configuration ?
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
尽管datax的功能还没有正式发布,但是我看了已提交的相关源码,发现仅支持4种读写方式,没法覆盖我当前的需求,于是设计了可自定义json配置的功能,大家有什么想法欢迎讨论。如图是工具栏选择datax后的侧边框,我想在红框的位置增加一个 tab 页,一页是当前的内容不变,另一页是文本框,用于用户自定义写json配置,同时需要对json格式做一个校验。两种方式二选一。
入参定义
请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig,值为0和1(数值类型),当customConfig等于0,为非自定义配置,其他参数没有变化,如下:
{"tasks":[{"params": {"customConfig":0, "targetTable":"test", "postStatements":[ "delete from test" ], "jobSpeedByte":0, "jobSpeedRecord":1000, "dtType":"MYSQL", "datasource":1, "dsType":"MYSQL", "datatarget":2, "sql":"select 1 as test from dual", "preStatements":[ "delete from test" ] } },{...}]}
当customConfig为1,为自定义json配置,只需传json字段(字符串类型)其他参数省略,如下:
{"tasks":[{"params":{ { "customConfig":1, "json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}" } }}] },{...}}
大家有什么想法欢迎讨论。
https://github.com/apache/incubator-dolphinscheduler/issues/2102
Re: add custom configuration Feature for datax 增加datax自定义配置功能
Posted by lidong dai <da...@gmail.com>.
good job
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
张世鸣 <36...@qq.com> 于2020年3月11日周三 下午7:19写道:
> https://github.com/apache/incubator-dolphinscheduler/issues/2102
>
>
>
> Although the datax plugin has not been officially released, I had read the
> relevant source code and found that it only supports a few reading and
> writing methods in the figure, which cannot cover my current needs. if it
> is possible to add the function of a custom datax configuration ?
>
> I will describe my ideas for implementing this feature.
> As shown below, I want to add a tab page , one page is the current content
> , and the other page is a text box for custom json configuration. At the
> same time, a check of the json format is required.
>
>
>
>
> 尽管datax的功能还没有正式发布,但是我看了已提交的相关源码,发现仅支持4种读写方式,没法覆盖我当前的需求,于是设计了可自定义json配置的功能,大家有什么想法欢迎讨论。如图是工具栏选择datax后的侧边框,我想在红框的位置增加一个
> tab 页,一页是当前的内容不变,另一页是文本框,用于用户自定义写json配置,同时需要对json格式做一个校验。两种方式二选一。
>
>
>
> 入参定义
>
>
> 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig,值为0和1(数值类型),当customConfig等于0,为非自定义配置,其他参数没有变化,如下:
> {"tasks":[{"params": {"customConfig":0, "targetTable":"test",
> "postStatements":[ "delete from test" ],
> "jobSpeedByte":0, "jobSpeedRecord":1000, "dtType":"MYSQL",
> "datasource":1, "dsType":"MYSQL", "datatarget":2,
> "sql":"select 1 as test from dual", "preStatements":[ "delete
> from test" ] } },{...}]}
> 当customConfig为1,为自定义json配置,只需传json字段(字符串类型)其他参数省略,如下:
> {"tasks":[{"params":{ { "customConfig":1,
> "json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT
> * from
> dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1
> = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}" }
> }}] },{...}}
>
>
>
>
> 大家有什么想法欢迎讨论。
> https://github.com/apache/incubator-dolphinscheduler/issues/2102