You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/12/15 14:37:16 UTC
[GitHub] [dolphinscheduler] devosend opened a new pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
devosend opened a new pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
close #7349
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
- Delete `database` task
- Extract get database info to `core/database` and adjust related operations of `sql task` and `produre task`
- Add `datax` task and `custom datax` task
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995638422
> i think database is not a kind of task. In the `datax` task, it needs two data sources, datasource and datatarget, and the keys in their parameters are also different. Inherited `Database` class can't meet the demand.
You are right! for now, database do not define any task in DS, and just a base class to get database information from java gateway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995481450
Could you please add some example about how to use this task type in directory `examples`? I think this task is a litter complex. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770390369
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+ """database object, get information about database.
+
+ You provider database_name contain connection information, it decisions which
+ database type and database instance would run task.
+ """
+
+ def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._database = {}
+ self.database_name = database_name
+ self[type_key] = self.database_type
+ self[database_key] = self.database_id
Review comment:
Database inherits the dict, which is convenient to use for params update like `params.update(datasource)`.
Because the database params related keys of `sql task` and `datax task` are different, the `type_key ` and `database_key ` need to be passed in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995853000
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995847180
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7437](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e168fe6) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/6cad119806360973db38690f4f4addf32d8e36c2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6cad119) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #7437 +/- ##
============================================
- Coverage 41.30% 41.29% -0.01%
- Complexity 3667 3668 +1
============================================
Files 634 634
Lines 26411 26411
Branches 2959 2959
============================================
- Hits 10908 10907 -1
Misses 14475 14475
- Partials 1028 1029 +1
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/plugin/registry/zookeeper/ZookeeperRegistry.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS9kb2xwaGluc2NoZWR1bGVyLXJlZ2lzdHJ5LXBsdWdpbnMvZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeS5qYXZh) | `48.64% <0.00%> (-7.21%)` | :arrow_down: |
| [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...inscheduler/common/task/sqoop/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3Rhc2svc3Fvb3AvU3Fvb3BQYXJhbWV0ZXJzLmphdmE=) | `74.00% <0.00%> (ø)` | |
| [...he/dolphinscheduler/common/enums/SqoopJobType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/7437/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL1Nxb29wSm9iVHlwZS5qYXZh) | `88.88% <0.00%> (+88.88%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6cad119...e168fe6](https://codecov.io/gh/apache/dolphinscheduler/pull/7437?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770244828
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+ """database object, get information about database.
+
+ You provider database_name contain connection information, it decisions which
+ database type and database instance would run task.
+ """
+
+ def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._database = {}
+ self.database_name = database_name
+ self[type_key] = self.database_type
+ self[database_key] = self.database_id
Review comment:
What `self[type_key]` use for? Do I miss something? I do not know this syntax before
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/datax.py
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Task datax."""
+
+from typing import Dict, List, Optional
+
+from pydolphinscheduler.constants import TaskType
+from pydolphinscheduler.core.database import Database
+from pydolphinscheduler.core.task import Task
+
+
+class CustomDataX(Task):
+ """Task CustomDatax object, declare behavior for custom DataX task to dolphinscheduler.
+
+ You provider json template for DataX, it can synchronize data according to the template you provided.
+ """
+
+ CUSTOM_CONFIG = 1
+
+ _task_custom_attr = {"custom_config", "json", "xms", "xmx"}
+
+ def __init__(
+ self,
+ name: str,
+ json: str,
+ xms: Optional[int] = 1,
+ xmx: Optional[int] = 1,
+ *args,
+ **kwargs
+ ):
+ super().__init__(name, TaskType.DATAX, *args, **kwargs)
+ self.custom_config = self.CUSTOM_CONFIG
+ self.json = json
+ self.xms = xms
+ self.xmx = xmx
+
+
+class DataX(Task):
+ """Task DataX object, declare behavior for DataX task to dolphinscheduler.
+
+ It should run database datax job in multiply sql link engine, such as:
+ - MySQL
+ - Oracle
+ - Postgresql
+ - SQLServer
+ You provider datasource_name and datatarget_name contain connection information, it decisions which
+ database type and database instance would synchronous data.
+ """
+
+ CUSTOM_CONFIG = 0
+
+ _task_custom_attr = {
+ "custom_config",
+ "sql",
+ "target_table",
+ "job_speed_byte",
+ "job_speed_record",
+ "pre_statements",
+ "post_statements",
+ "xms",
+ "xmx",
+ }
+
+ def __init__(
+ self,
+ name: str,
+ datasource_name: str,
+ datatarget_name: str,
+ sql: str,
+ target_table: str,
+ job_speed_byte: Optional[int] = 0,
+ job_speed_record: Optional[int] = 1000,
+ pre_statements: Optional[List[str]] = [],
+ post_statements: Optional[List[str]] = [],
Review comment:
Use mutable object in function `__init__` init is not the best practices and it would case value overwrite, you could change and use
```suggestion
pre_statements: Optional[List[str]] = None,
post_statements: Optional[List[str]] = None,
```
and in L103-104
```py
self.pre_statements = pre_statements or []
self.post_statements = post_statements or []
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995473789
BTW, please solve the conflicts, Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995628995
i think database is not a kind of task. In the `datax` task, it needs two data sources, datasource and datatarget, and the keys in their parameters are also different. Inherited `Database` class can't meet the demand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770390991
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/datax.py
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Task datax."""
+
+from typing import Dict, List, Optional
+
+from pydolphinscheduler.constants import TaskType
+from pydolphinscheduler.core.database import Database
+from pydolphinscheduler.core.task import Task
+
+
+class CustomDataX(Task):
+ """Task CustomDatax object, declare behavior for custom DataX task to dolphinscheduler.
+
+ You provider json template for DataX, it can synchronize data according to the template you provided.
+ """
+
+ CUSTOM_CONFIG = 1
+
+ _task_custom_attr = {"custom_config", "json", "xms", "xmx"}
+
+ def __init__(
+ self,
+ name: str,
+ json: str,
+ xms: Optional[int] = 1,
+ xmx: Optional[int] = 1,
+ *args,
+ **kwargs
+ ):
+ super().__init__(name, TaskType.DATAX, *args, **kwargs)
+ self.custom_config = self.CUSTOM_CONFIG
+ self.json = json
+ self.xms = xms
+ self.xmx = xmx
+
+
+class DataX(Task):
+ """Task DataX object, declare behavior for DataX task to dolphinscheduler.
+
+ It should run database datax job in multiply sql link engine, such as:
+ - MySQL
+ - Oracle
+ - Postgresql
+ - SQLServer
+ You provider datasource_name and datatarget_name contain connection information, it decisions which
+ database type and database instance would synchronous data.
+ """
+
+ CUSTOM_CONFIG = 0
+
+ _task_custom_attr = {
+ "custom_config",
+ "sql",
+ "target_table",
+ "job_speed_byte",
+ "job_speed_record",
+ "pre_statements",
+ "post_statements",
+ "xms",
+ "xmx",
+ }
+
+ def __init__(
+ self,
+ name: str,
+ datasource_name: str,
+ datatarget_name: str,
+ sql: str,
+ target_table: str,
+ job_speed_byte: Optional[int] = 0,
+ job_speed_record: Optional[int] = 1000,
+ pre_statements: Optional[List[str]] = [],
+ post_statements: Optional[List[str]] = [],
Review comment:
you are right. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995621601
ok, I will submit an example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-994869194
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend edited a comment on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend edited a comment on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995621601
ok, I will submit an example. @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie merged pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie merged pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995473511
Hi @devosend I notice that you move class `database` from task to core, is there any reason or benefit about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
devosend commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995630112
Do you have any other suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-995638936
You make a good change 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#discussion_r770405396
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module database."""
+
+from typing import Dict
+
+from pydolphinscheduler.java_gateway import launch_gateway
+
+
+class Database(dict):
+ """database object, get information about database.
+
+ You provider database_name contain connection information, it decisions which
+ database type and database instance would run task.
+ """
+
+ def __init__(self, database_name: str, type_key, database_key, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._database = {}
+ self.database_name = database_name
+ self[type_key] = self.database_type
+ self[database_key] = self.database_id
Review comment:
Oh, I get it, from dict. Good job Dongkai 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #7437: [Feature-7349] [Python]Add workflow as code task type datax
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #7437:
URL: https://github.com/apache/dolphinscheduler/pull/7437#issuecomment-994869194
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=7437&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=7437&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org