You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/01/05 03:52:19 UTC
[GitHub] [dolphinscheduler] zhongjiajie opened a new pull request #7801: [python] Enhance task datax example
zhongjiajie opened a new pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801
* Add full example for `CustomDataX.json`
* Add comment about datasource need to exists.
close: #7800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778626502
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
Please confirm whether there is any problem with this. @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005363828
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7801](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2485a95) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/120c1755a15fcf78df303a67edfa87db0a7ff495?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (120c175) will **decrease** coverage by `0.35%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #7801 +/- ##
============================================
- Coverage 41.18% 40.83% -0.36%
+ Complexity 3669 3666 -3
============================================
Files 641 637 -4
Lines 26525 26783 +258
Branches 2964 3017 +53
============================================
+ Hits 10924 10936 +12
- Misses 14575 14797 +222
- Partials 1026 1050 +24
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Ξ | |
|---|---|---|
| [...he/dolphinscheduler/common/enums/SqoopJobType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL1Nxb29wSm9iVHlwZS5qYXZh) | `0.00% <0.00%> (-88.89%)` | :arrow_down: |
| [...inscheduler/server/log/LoggerRequestProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1sb2ctc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZlci9sb2cvTG9nZ2VyUmVxdWVzdFByb2Nlc3Nvci5qYXZh) | `20.31% <0.00%> (-39.07%)` | :arrow_down: |
| [...e/dolphinscheduler/api/aspect/AccessLogAspect.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2FzcGVjdC9BY2Nlc3NMb2dBc3BlY3QuamF2YQ==) | `67.24% <0.00%> (-13.72%)` | :arrow_down: |
| [...er/api/service/impl/TaskDefinitionServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9UYXNrRGVmaW5pdGlvblNlcnZpY2VJbXBsLmphdmE=) | `26.13% <0.00%> (-5.70%)` | :arrow_down: |
| [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...heduler/api/service/impl/TaskGroupServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9UYXNrR3JvdXBTZXJ2aWNlSW1wbC5qYXZh) | `60.17% <0.00%> (-3.83%)` | :arrow_down: |
| [...org/apache/dolphinscheduler/remote/utils/Host.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL3V0aWxzL0hvc3QuamF2YQ==) | `37.77% <0.00%> (-2.23%)` | :arrow_down: |
| [...inscheduler/common/task/sqoop/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3Rhc2svc3Fvb3AvU3Fvb3BQYXJhbWV0ZXJzLmphdmE=) | `74.00% <0.00%> (-2.00%)` | :arrow_down: |
| [...e/dolphinscheduler/remote/NettyRemotingClient.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL05ldHR5UmVtb3RpbmdDbGllbnQuamF2YQ==) | `52.11% <0.00%> (-1.41%)` | :arrow_down: |
| [...inscheduler/server/master/config/MasterConfig.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9jb25maWcvTWFzdGVyQ29uZmlnLmphdmE=) | `69.56% <0.00%> (-0.44%)` | :arrow_down: |
| ... and [39 more](https://codecov.io/gh/apache/dolphinscheduler/pull/7801/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Ξ = absolute <relative> (impact)`, `ΓΈ = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [120c175...2485a95](https://codecov.io/gh/apache/dolphinscheduler/pull/7801?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005367087
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778744222
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
> are there any error tips
My laptop is lack of resources to run full standalone server, it only could run api-server π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778762251
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
Thank!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778633263
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
Ok, I would take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005362108
@devosend PTAL if you have time, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778621989
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
I failed to run this task. Error message:
So I think you should add `setting` key after `content` key.
```
"setting":{
"errorLimit":{
"percentage":0,
"record":0
},
"speed":{
"channel":1,
"record":1000
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005367087
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778743464
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
Did you test this PR code? If you did and it could run datax task, make we could directly merge this PR now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005361957
If `CustomDataX.json` without specific field, would failed in `checkTaskDefinitionList` https://github.com/apache/dolphinscheduler/blob/01a2b9684a4c6478ca4aea74b2307558c57468a7/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionServiceImpl.java#L210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778742240
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
I run with web ui, t's correct. are there any error tips?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005560309
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7801&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7801&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs merged pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778696407
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
But I have one more thing to said, I try to start datax jobs in my local environment but failed π π π . So I can not reproduce the case you said above https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778621989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#issuecomment-1005362108
@devosend PTAL if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778694453
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
> Please confirm whether there is any problem with this. @zhongjiajie
@devosend I add the required attribute `setting.speed` for json define. see https://github.com/alibaba/DataX/blob/master/userGuid.md#quick-start , I also add some optional setting basic on your advice above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7801: [python] Enhance task datax example
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7801:
URL: https://github.com/apache/dolphinscheduler/pull/7801#discussion_r778756219
##########
File path: dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py
##########
@@ -29,14 +29,67 @@
from pydolphinscheduler.tasks.datax import CustomDataX, DataX
# datax json template
-JSON_TEMPLATE = ""
+JSON_TEMPLATE = {
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "mysqlreader",
+ "parameter": {
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name",
+ "code",
+ "description"
+ ],
+ "splitPk": "id",
+ "connection": [
+ {
+ "table": [
+ "source_table"
+ ],
+ "jdbcUrl": [
+ "jdbc:mysql://127.0.0.1:3306/source_db"
+ ]
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "mysqlwriter",
+ "parameter": {
+ "writeMode": "insert",
+ "username": "usr",
+ "password": "pwd",
+ "column": [
+ "id",
+ "name"
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/target_db",
+ "table": [
+ "target_table"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ]
Review comment:
I run it in my local. it's ok. Merge it please. @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org