You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/12/15 14:37:16 UTC

[GitHub] [dolphinscheduler] devosend opened a new pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

devosend opened a new pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437


   <!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
   
   ## Purpose of the pull request
   close #7349 
   <!--(For example: This pull request adds checkstyle plugin).-->
   
   ## Brief change log
   - Delete `database` task
   - Extract get database info to `core/database` and adjust related operations of `sql task` and `produre task`
   - Add `datax` task and `custom datax` task
   
   <!--*(for example:)*
     - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   ## Verify this pull request
   
   <!--*(Please pick either of the following options)*-->
   
   This pull request is code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   <!--*(example:)*
     - *Added dolphinscheduler-dao tests for end-to-end.*
     - *Added CronUtilsTest to verify the change.*
     - *Manually verified the change by testing locally.* -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995638422


   > i think database is not a kind of task. In the `datax` task, it needs two data sources, datasource and datatarget, and the keys in their parameters are also different. Inherited `Database` class can't meet the demand.
   
   You are right! for now, database do not define any task in DS, and just a base class to get database information from java gateway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995481450


   Could you please add some example about how to use this task type in directory `examples`? I think this task is a litter complex. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770390369



##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+    """database object, get information about database.
+
+    You provider database_name contain connection information, it decisions which
+    database type and database instance would run task.
+    """
+
+    def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._database = {}
+        self.database_name = database_name
+        self[type_key] = self.database_type
+        self[database_key] = self.database_id

Review comment:
       Database inherits the dict, which is convenient to use for params update like `params.update(datasource)`.
   Because the database params related keys of `sql task` and `datax task` are different, the `type_key ` and `database_key ` need to be passed in.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995853000


   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; ![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995847180


   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7437](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e168fe6) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/6cad119806360973db38690f4f4addf32d8e36c2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cad119) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/graphs/tree.svg?width=650&height=150&src=pr&token=bv9iXXRLi9&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev    #7437      +/-   ##
   ============================================
   - Coverage     41.30%   41.29%   -0.01%     
   - Complexity     3667     3668       +1     
   ============================================
     Files           634      634              
     Lines         26411    26411              
     Branches       2959     2959              
   ============================================
   - Hits          10908    10907       -1     
     Misses        14475    14475              
   - Partials       1028     1029       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...r/plugin/registry/zookeeper/ZookeeperRegistry.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS9kb2xwaGluc2NoZWR1bGVyLXJlZ2lzdHJ5LXBsdWdpbnMvZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeS5qYXZh) | `48.64% <0.00%> (-7.21%)` | :arrow_down: |
   | [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
   | [...inscheduler/common/task/sqoop/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3Rhc2svc3Fvb3AvU3Fvb3BQYXJhbWV0ZXJzLmphdmE=) | `74.00% <0.00%> (ø)` | |
   | [...he/dolphinscheduler/common/enums/SqoopJobType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL1Nxb29wSm9iVHlwZS5qYXZh) | `88.88% <0.00%> (+88.88%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cad119...e168fe6](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770244828



##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+    """database object, get information about database.
+
+    You provider database_name contain connection information, it decisions which
+    database type and database instance would run task.
+    """
+
+    def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._database = {}
+        self.database_name = database_name
+        self[type_key] = self.database_type
+        self[database_key] = self.database_id

Review comment:
       What `self[type_key]` use for? Do I miss something? I do not know this syntax before

##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/datax.py
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Task datax."""
+
+from typing import Dict, List, Optional
+
+from pydolphinscheduler.constants import TaskType
+from pydolphinscheduler.core.database import Database
+from pydolphinscheduler.core.task import Task
+
+
+class CustomDataX(Task):
+    """Task CustomDatax object, declare behavior for custom DataX task to dolphinscheduler.
+
+    You provider json template for DataX, it can synchronize data according to the template you provided.
+    """
+
+    CUSTOM_CONFIG = 1
+
+    _task_custom_attr = {"custom_config", "json", "xms", "xmx"}
+
+    def __init__(
+        self,
+        name: str,
+        json: str,
+        xms: Optional[int] = 1,
+        xmx: Optional[int] = 1,
+        *args,
+        **kwargs
+    ):
+        super().__init__(name, TaskType.DATAX, *args, **kwargs)
+        self.custom_config = self.CUSTOM_CONFIG
+        self.json = json
+        self.xms = xms
+        self.xmx = xmx
+
+
+class DataX(Task):
+    """Task DataX object, declare behavior for DataX task to dolphinscheduler.
+
+    It should run database datax job in multiply sql link engine, such as:
+    - MySQL
+    - Oracle
+    - Postgresql
+    - SQLServer
+    You provider datasource_name and datatarget_name contain connection information, it decisions which
+    database type and database instance would synchronous data.
+    """
+
+    CUSTOM_CONFIG = 0
+
+    _task_custom_attr = {
+        "custom_config",
+        "sql",
+        "target_table",
+        "job_speed_byte",
+        "job_speed_record",
+        "pre_statements",
+        "post_statements",
+        "xms",
+        "xmx",
+    }
+
+    def __init__(
+        self,
+        name: str,
+        datasource_name: str,
+        datatarget_name: str,
+        sql: str,
+        target_table: str,
+        job_speed_byte: Optional[int] = 0,
+        job_speed_record: Optional[int] = 1000,
+        pre_statements: Optional[List[str]] = [],
+        post_statements: Optional[List[str]] = [],

Review comment:
       Use mutable object in function `__init__` init is not the best practices and it would case value overwrite, you could change and use 
   ```suggestion
           pre_statements: Optional[List[str]] = None,
           post_statements: Optional[List[str]] = None,
   ```
   and in L103-104
   ```py
           self.pre_statements = pre_statements or []
           self.post_statements = post_statements or []
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995473789


   BTW, please solve the conflicts, Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995628995


   i think database is not a kind of task.  In the `datax` task, it needs two data sources, datasource and datatarget, and the keys in their parameters are also different. Inherited `Database` class can't meet the demand.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770390991



##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/datax.py
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Task datax."""
+
+from typing import Dict, List, Optional
+
+from pydolphinscheduler.constants import TaskType
+from pydolphinscheduler.core.database import Database
+from pydolphinscheduler.core.task import Task
+
+
+class CustomDataX(Task):
+    """Task CustomDatax object, declare behavior for custom DataX task to dolphinscheduler.
+
+    You provider json template for DataX, it can synchronize data according to the template you provided.
+    """
+
+    CUSTOM_CONFIG = 1
+
+    _task_custom_attr = {"custom_config", "json", "xms", "xmx"}
+
+    def __init__(
+        self,
+        name: str,
+        json: str,
+        xms: Optional[int] = 1,
+        xmx: Optional[int] = 1,
+        *args,
+        **kwargs
+    ):
+        super().__init__(name, TaskType.DATAX, *args, **kwargs)
+        self.custom_config = self.CUSTOM_CONFIG
+        self.json = json
+        self.xms = xms
+        self.xmx = xmx
+
+
+class DataX(Task):
+    """Task DataX object, declare behavior for DataX task to dolphinscheduler.
+
+    It should run database datax job in multiply sql link engine, such as:
+    - MySQL
+    - Oracle
+    - Postgresql
+    - SQLServer
+    You provider datasource_name and datatarget_name contain connection information, it decisions which
+    database type and database instance would synchronous data.
+    """
+
+    CUSTOM_CONFIG = 0
+
+    _task_custom_attr = {
+        "custom_config",
+        "sql",
+        "target_table",
+        "job_speed_byte",
+        "job_speed_record",
+        "pre_statements",
+        "post_statements",
+        "xms",
+        "xmx",
+    }
+
+    def __init__(
+        self,
+        name: str,
+        datasource_name: str,
+        datatarget_name: str,
+        sql: str,
+        target_table: str,
+        job_speed_byte: Optional[int] = 0,
+        job_speed_record: Optional[int] = 1000,
+        pre_statements: Optional[List[str]] = [],
+        post_statements: Optional[List[str]] = [],

Review comment:
       you are right. thanks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995621601


   ok, I will submit an example.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-994869194


   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; ![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend edited a comment on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend edited a comment on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995621601


   ok, I will submit an example. @zhongjiajie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie merged pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie merged pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995473511


   Hi @devosend I notice that you move class `database` from task to core, is there any reason or benefit about that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995630112


   Do you have any other suggestions?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995638936


   You make a good change 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770405396



##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+    """database object, get information about database.
+
+    You provider database_name contain connection information, it decisions which
+    database type and database instance would run task.
+    """
+
+    def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._database = {}
+        self.database_name = database_name
+        self[type_key] = self.database_type
+        self[database_key] = self.database_id

Review comment:
       Oh, I get it, from dict. Good job Dongkai 👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-994869194


   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; ![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org