You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/03/06 09:35:54 UTC
[GitHub] [incubator-dolphinscheduler] simon824 opened a new issue #2102:
[Feature]add custom configuration Feature for datax
simon824 opened a new issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102
Although the datax plugin has not been officially released, but I looked at the relevant source code and found that it only supports a few reading and writing methods in the figure, which cannot cover my current needs. if it is possible to add the function of a custom datax configuration ? If this function makes sense, I am glad to implement it.
尽管datax的功能还没有正式发布，但是我看了已提交的相关源码，发现仅支持图中的几种读写方式，没法覆盖我当前的需求，是否可以增加自定义datax配置文件的功能，如果这个功能有意义，我很乐意去实现它。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] Rubik-W commented on issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
Rubik-W commented on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597583298
With a full amount of custom configuration, the connection parameters of the database will be exposed. This requires attention.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 commented on issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
如图，我想在红框的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，同时需要对json格式做一个校验。两种方式二选一。
## 入参定义
- 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
- 当customConfig为1，为自定义json配置，只需传json字段（字符串类型）其他参数省略，如下：
```
{"tasks":[{"params":{
{
"customConfig":1,
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] break60 commented on issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
break60 commented on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-602222570
#2218
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 closed issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 closed issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
上图是当前工具栏选择datax后的侧边框，我想在红圈的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，底部增加自定参数，如下图，同时需要前端对json格式做一个校验。
## 入参定义
1. 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
增加如下参数
- customConfig 是否为自定义配置，1为是，0为否
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
2. 当customConfig为1，为自定义json配置，需传json字段（字符串类型），如果指定了自定义参数则增加localParams字段，如下：
增加如下参数：
- customConfig 是否为自定义配置，1为是，0为否
- json 自定义配置内容
- localParams 自定义参数
```
{"tasks":[{"params":{
{
"customConfig":1,
"localParams":[
{
"prop":"test",
"direct":"IN",
"type":"VARCHAR",
"value":"38294729"
}
],
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 opened a new issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 opened a new issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102
Although the datax plugin has not been officially released, I had read the relevant source code and found that it only supports a few reading and writing methods in the figure, which cannot cover my current needs. if it is possible to add the function of a custom datax configuration ? If this function makes sense, I am glad to implement it.
尽管datax的功能还没有正式发布，但是我看了已提交的相关源码，发现仅支持图中的几种读写方式，没法覆盖我当前的需求，是否可以增加自定义datax配置文件的功能，如果这个功能有意义，我很乐意去实现它。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
如图是工具栏选择datax后的侧边框，我想在红框的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，同时需要对json格式做一个校验。两种方式二选一。
## 入参定义
- 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
- 当customConfig为1，为自定义json配置，只需传json字段（字符串类型）其他参数省略，如下：
```
{"tasks":[{"params":{
{
"customConfig":1,
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
上图是当前工具栏选择datax后的侧边框，我想在红圈的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，底部增加自定参数，如下图，同时需要前端对json格式做一个校验。
## 入参定义
1. 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
增加如下参数
- customConfig 是否为自定义配置，1为是，0为否
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
2. 当customConfig为1，为自定义json配置，需传json字段（字符串类型），如果指定了自定义参数则增加localParams字段，如下：
增加如下参数：
- customConfig 是否为自定义配置，1为是，0为否
- json 自定义配置内容
- localParams 自定义参数
```
{"tasks":[{"params":{
{
"customConfig":1,
"localParams":[
{
"prop":"test",
"value":"38294729"
}
],
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 closed issue #2102:
[Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 closed issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
上图是当前工具栏选择datax后的侧边框，我想在红圈的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，底部增加自定参数，如下图，同时需要前端对json格式做一个校验。
## 入参定义
- 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
- 当customConfig为1，为自定义json配置，需传json字段（字符串类型），如果指定了自定义参数则增加localParams字段，如下：
```
{"tasks":[{"params":{
{
"customConfig":1,
"localParams":[
{
"prop":"test",
"value":"38294729"
}
],
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
- customConfig 是否为自定义配置
- json 自定义配置内容
- localParams 自定义参数
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
如图是工具栏选择datax后的侧边框，我想在红框的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，底部增加自定参数，如下图，同时需要前端对json格式做一个校验。
## 入参定义
- 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
- 当customConfig为1，为自定义json配置，需传json字段（字符串类型），如果指定了自定义参数则增加localParams字段，如下：
```
{"tasks":[{"params":{
{
"customConfig":1,
"localParams":[
{
"prop":"test",
"direct":"IN",
"type":"VARCHAR",
"value":"38294729"
}
],
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] simon824 edited a comment on issue
#2102: [Feature]add custom configuration Feature for datax
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on issue #2102: [Feature]add custom configuration Feature for datax
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2102#issuecomment-597538608
I will describe my ideas for implementing this feature.
As shown below, I want to add a tab page , one page is the current content , and the other page is a text box for custom json configuration. At the same time, a check of the json format is required.
我来详细说明下我的想法，以方便和前端的同学协作完成功能的开发。
如图是当前工具栏选择datax后的侧边框，我想在红圈的位置增加一个 tab 页，一页是当前的内容不变，另一页是文本框，用于用户自定义写json配置，底部增加自定参数，如下图，同时需要前端对json格式做一个校验。
## 入参定义
- 请求参数processDefinitionJson的tasks列表里的params字段的内容增加customConfig，值为0和1（数值类型），当customConfig等于0，为非自定义配置，其他参数没有变化，如下：
```
{"tasks":[{"params":
{"customConfig":0,
"targetTable":"test",
"postStatements":[
"delete from test"
],
"jobSpeedByte":0,
"jobSpeedRecord":1000,
"dtType":"MYSQL",
"datasource":1,
"dsType":"MYSQL",
"datatarget":2,
"sql":"select 1 as test from dual",
"preStatements":[
"delete from test"
]
}
},{...}]}
```
- 当customConfig为1，为自定义json配置，需传json字段（字符串类型），如果指定了自定义参数则增加localParams字段，如下：
```
{"tasks":[{"params":{
{
"customConfig":1,
"localParams":[
{
"prop":"test",
"direct":"IN",
"type":"VARCHAR",
"value":"38294729"
}
],
"json":"{"job":{"setting":{"speed":{"byte":1048576},"errorLimit":{"record":0,"percentage":0.02}},"content":[{"reader":{"name":"rdbmsreader","parameter":{"username":"xxx","password":"xxx","column":["id","name"],"splitPk":"pk","connection":[{"querySql":["SELECT * from dual"],"jdbcUrl":["jdbc:dm://ip:port/database"]}],"fetchSize":1024,"where":"1 = 1"}},"writer":{"name":"streamwriter","parameter":{"print":true}}}]}}"
}
}}]
},{...}}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services