You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/20 11:55:12 UTC
[GitHub] [dolphinscheduler] hiSandog opened a new pull request, #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
hiSandog opened a new pull request, #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515
Now when I define a Python task that requires importing resources via python-gate.
I need to check the id of the resource in the database first.
And then I write code that looks something like this.
```
test_task = Python(
name="测试任务",
resource_list=[{"id": 64}, {"id": 65}, {"id": 66}, {"id": 67}, {"id": 68}],
code=base_info_py_code
)
```
resource_list should be allowed to be defined using the resource full-name ,like this
```
test_task = Python(
name="测试任务",
resource_list=[{"resourceName": "/abce/aa.py"}, {"resourceName": "/abce/bb.py"}],
code=base_info_py_code
)
```
close: #10486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515#issuecomment-1160413720
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10515)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515#issuecomment-1160388948
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10515)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515#discussion_r903492117
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/python.py:
##########
@@ -100,3 +108,21 @@ def raw_script(self) -> str:
raise PyDSParamException(
"Parameter definition do not support % for now.", type(self.definition)
)
+
+ @property
+ def resource_list(self) -> List:
+ """Get python task define attribute `resource_list`."""
+ if super().resource_list is not None and len(super().resource_list) > 0:
+ for resource in super().resource_list:
+ if (
+ resource.get("id") is None
+ and resource.get("resourceName") is not None
+ ):
+ if resource.get("resourceType") is not None:
+ resource_type = resource.get("resourceType")
+ else:
+ resource_type = "FILE"
+ resource["id"] = self.query_resource(
+ resource_type, resource.get("resourceName")
+ ).get("id")
+ return super().resource_list
Review Comment:
I think we should better put this to `pydolphinscheduler.core.task` instead of python.py. because I know we also support it in shell task. and in the `core.task` we can add some code like
```py
@property
def resource_list(self) -> List:
"""Get attribute resource_list."""
resources = set()
for resource in self._resource_list:
resources.add(query_resource(resource_type, resource))
return [{"id": r.get("id")} for r in resources]
@resource_list.setter
def resource_list(self, val) -> None:
"""Get attribute resource_list."""
return self._resource_list = val
```
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/python.py:
##########
@@ -100,3 +108,21 @@ def raw_script(self) -> str:
raise PyDSParamException(
"Parameter definition do not support % for now.", type(self.definition)
)
+
+ @property
+ def resource_list(self) -> List:
+ """Get python task define attribute `resource_list`."""
+ if super().resource_list is not None and len(super().resource_list) > 0:
+ for resource in super().resource_list:
+ if (
+ resource.get("id") is None
+ and resource.get("resourceName") is not None
+ ):
+ if resource.get("resourceType") is not None:
+ resource_type = resource.get("resourceType")
+ else:
+ resource_type = "FILE"
+ resource["id"] = self.query_resource(
+ resource_type, resource.get("resourceName")
+ ).get("id")
+ return super().resource_list
Review Comment:
BTW, why we need resource type to query the resources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hiSandog closed pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
hiSandog closed pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
URL: https://github.com/apache/dolphinscheduler/pull/10515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515#issuecomment-1160509074
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10515)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10515&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10515&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10515: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10515:
URL: https://github.com/apache/dolphinscheduler/pull/10515#discussion_r903495127
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/python.py:
##########
@@ -100,3 +108,21 @@ def raw_script(self) -> str:
raise PyDSParamException(
"Parameter definition do not support % for now.", type(self.definition)
)
+
+ @property
+ def resource_list(self) -> List:
+ """Get python task define attribute `resource_list`."""
+ if super().resource_list is not None and len(super().resource_list) > 0:
+ for resource in super().resource_list:
+ if (
+ resource.get("id") is None
+ and resource.get("resourceName") is not None
+ ):
+ if resource.get("resourceType") is not None:
+ resource_type = resource.get("resourceType")
+ else:
+ resource_type = "FILE"
+ resource["id"] = self.query_resource(
+ resource_type, resource.get("resourceName")
+ ).get("id")
+ return super().resource_list
Review Comment:
And if we finish https://github.com/apache/dolphinscheduler/pull/10515#discussion_r903492117, user can not pass `[{"resourceName": "/abce/aa.py"}, {"resourceName": "/abce/bb.py"}]` to `resource_list` anymore, they can simply pass `["/abce/aa.py", ""/abce/bb.py""]` to it, and we will autofill the `resourceName` keyword adn return with `id`. WDTY?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org