You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/22 11:44:48 UTC
[GitHub] [dolphinscheduler] hiSandog opened a new pull request, #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
hiSandog opened a new pull request, #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551
Now when I define a Python task that requires importing resources via python-gate.
I need to check the id of the resource in the database first.
And then I write code that looks something like this.
```
test_task = Python(
name="测试任务",
resource_list=[{"id": 64}, {"id": 65}, {"id": 66}, {"id": 67}, {"id": 68}],
code=base_info_py_code
)
```
resource_list should be allowed to be defined using the resource full-name ,like this
```
test_task = Python(
name="测试任务",
resource_list=["/abce/aa.py", "/abce/bb.py"],
code=base_info_py_code
)
```
close: #10486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1163979499
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1163017581
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1164425388
Waiting for CI, and I will merge the code when it pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#discussion_r905038210
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/task.py:
##########
@@ -170,6 +171,22 @@ def process_definition(self, process_definition: Optional[ProcessDefinition]):
"""Set attribute process_definition."""
self._process_definition = process_definition
+ @property
+ def resource_list(self) -> List:
+ """Get task define attribute `resource_list`."""
+ resources = set()
+ for resource in self._resource_list:
+ if type(resource) == str:
+ resources.add(self.query_resource(resource).get(ResourceKey.ID))
+ elif type(resource) == dict and resource.get(ResourceKey.ID) is not None:
+ logger.warning(
+ """Resource_list should be defined using List[str] with resource paths,
Review Comment:
```suggestion
"""`resource_list` should be defined using List[str] with resource paths,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [python] Support using full-name definitions for resources
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1164453916
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1163119275
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1164016725
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1163038437
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10551)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10551&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list) [69.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10551&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1163011424
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10551](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b6e765) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/9345c16f3f1e5232041cd65ddee40e84602d1a62?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9345c16) will **increase** coverage by `0.00%`.
> The diff coverage is `63.63%`.
```diff
@@ Coverage Diff @@
## dev #10551 +/- ##
=========================================
Coverage 40.88% 40.88%
- Complexity 4867 4868 +1
=========================================
Files 893 893
Lines 36118 36129 +11
Branches 3992 3993 +1
=========================================
+ Hits 14766 14772 +6
- Misses 19890 19896 +6
+ Partials 1462 1461 -1
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/dolphinscheduler/api/python/PythonGateway.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3B5dGhvbi9QeXRob25HYXRld2F5LmphdmE=) | `19.43% <63.63%> (+2.43%)` | :arrow_up: |
| [...r/plugin/task/sqoop/parameter/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Fvb3Avc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3Fvb3AvcGFyYW1ldGVyL1Nxb29wUGFyYW1ldGVycy5qYXZh) | `53.33% <0.00%> (-1.34%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9345c16...3b6e765](https://codecov.io/gh/apache/dolphinscheduler/pull/10551?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10551: [python] Support using full-name definitions for resources
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#issuecomment-1164519969
Merge and thanks @hiSandog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie merged pull request #10551: [python] Support using full-name definitions for resources
Posted by GitBox <gi...@apache.org>.
zhongjiajie merged PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10551: [Feature-10486][Python] Python-gate support the use of full-name definitions for resources.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10551:
URL: https://github.com/apache/dolphinscheduler/pull/10551#discussion_r904480035
##########
dolphinscheduler-python/pydolphinscheduler/tests/core/test_task.py:
##########
@@ -47,14 +47,14 @@
(
{
"local_params": ["foo", "bar"],
- "resource_list": ["foo", "bar"],
+ "resource_list": [{"id": 1}, {"id": 2}],
"dependence": {"foo", "bar"},
"wait_start_timeout": {"foo", "bar"},
"condition_result": {"foo": ["bar"]},
},
{
"localParams": ["foo", "bar"],
- "resourceList": ["foo", "bar"],
+ "resourceList": [{"id": 1}, {"id": 2}],
Review Comment:
I find out we just add style `[{"key": val}, {"key": val}]` for test, can we also add `["foo", "bar"]` type resource for test and mock the function `query_resource` return values? I am not sure whether we can add this case to this function or not, but if we can not, please add new test function to cover it
##########
dolphinscheduler-python/pydolphinscheduler/tests/core/test_task.py:
##########
@@ -241,3 +241,34 @@ def test_add_duplicate(caplog):
re.findall("already in process definition", caplog.text),
]
)
+
+
+@pytest.mark.parametrize(
+ "resources, expect",
+ [
+ (
+ ["/dev/test.py"],
+ [{"id": 1}],
+ ),
+ (
+ ["/dev/test.py", {"id": 2}],
+ [{"id": 1}, {"id": 2}],
+ ),
+ ],
+)
+@patch(
+ "pydolphinscheduler.core.task.Task.gen_code_and_version",
+ return_value=(123, 1),
+)
+@patch(
+ "pydolphinscheduler.core.task.Task.query_resource",
+ return_value=({"id": 1, "name": "/dev/test.py"}),
+)
+def test_python_resource_list(mock_code_version, mock_resource, resources, expect):
Review Comment:
Thanks for adding this tests
##########
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/python/PythonGatewayTest.java:
##########
@@ -83,6 +92,37 @@ public void testGetDependentInfo() {
Assert.assertEquals((long) result.get("taskDefinitionCode"), taskDefinition.getCode());
}
+
+ @Test
+ public void testQueryResourcesFileInfo() {
Review Comment:
Great, thanks for adding tests for it
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/task.py:
##########
@@ -170,6 +170,17 @@ def process_definition(self, process_definition: Optional[ProcessDefinition]):
"""Set attribute process_definition."""
self._process_definition = process_definition
+ @property
+ def resource_list(self) -> List:
+ """Get task define attribute `resource_list`."""
+ resources = list()
+ for resource in self._resource_list:
+ if type(resource) == str:
+ resources.append(self.query_resource(resource))
+ elif type(resource) == dict and resource.get("id") is not None:
+ resources.append(resource)
Review Comment:
can we use `warnings.warn` to alert users this feat will is deprecation, should be use `List[str]` instead, and will be remove in version `3.2.0`?
##########
dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/task.py:
##########
@@ -170,6 +170,17 @@ def process_definition(self, process_definition: Optional[ProcessDefinition]):
"""Set attribute process_definition."""
self._process_definition = process_definition
+ @property
+ def resource_list(self) -> List:
+ """Get task define attribute `resource_list`."""
+ resources = list()
+ for resource in self._resource_list:
+ if type(resource) == str:
+ resources.append(self.query_resource(resource))
+ elif type(resource) == dict and resource.get("id") is not None:
+ resources.append(resource)
+ return [{"id": r.get("id")} for r in resources]
Review Comment:
could you add `"id"` to files constants? https://github.com/apache/dolphinscheduler/blob/80ebe4a33431c88c7514b3ad0d9ffdf645e1bba2/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/constants.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org