You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/09/21 01:54:45 UTC
[GitHub] [dolphinscheduler] EricPyZhou opened a new pull request, #12076: [Feature-10495][Resource Center] Resource Center Refactor
EricPyZhou opened a new pull request, #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
https://github.com/apache/dolphinscheduler/issues/10495
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r976190640
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -790,6 +790,22 @@ CREATE TABLE `t_ds_resources` (
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_relation_resources_task`;
+CREATE TABLE `t_ds_relation_resources_task` (
+ `id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
+ `full_name` varchar(255) DEFAULT NULL,
+ `type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `t_ds_relation_resources_task_un` (`full_name`,`type`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3
Review Comment:
I think this table is about resources, not the relation between task and resources.
Why not create this table like this, and then we can remove the `resource_ids_new` field in `t_ds_task_definition`
```suggestion
CREATE TABLE `t_ds_relation_resources_task` (
`id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
`task_id` int(11) DEFAULT NULL COMMENT 'task id',
`full_name` varchar(255) DEFAULT NULL,
`type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
PRIMARY KEY (`id`),
UNIQUE KEY `t_ds_relation_resources_task_un` (`task_id`, `full_name`,`type`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1274042117
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996369354
##########
dolphinscheduler-ui/src/views/projects/task/components/node/types.ts:
##########
@@ -91,7 +91,9 @@ interface ISwitchResult {
}
interface ISourceItem {
- id: number
+ id?: number,
+ resourceName: string,
+ res?: string
Review Comment:
> what's the meaning of `res`? Would it be better to use a full word？
ya I agree that 'res' is not clear. It is kinda hard to change it because 'res' corresponds to 'res' field in the backend. For a temporal fix, I put a comment box there to illustrate what 'resourceName' and 'res' are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS merged pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
SbloodyS merged PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1287991731
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [89 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1291376476
> Is it need to update doc? Please check it.
This PR does not change the way users use it, which is insensitive to the user experience. I think we don't need to change the docs. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276345436
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-code-scanning[bot] commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996627893
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -112,14 +120,20 @@
return result;
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e) {
+ logger.error("AmazonServiceException when checking resource: " + fullName);
+ }
+
+ if (!existResource) {
+ logger.error("resource full name {} is not exist", fullName);
Review Comment:
## Logging should not be vulnerable to injection attacks
<!--SONAR_ISSUE_KEY:AYPkb5rB3zqccheRFO5r-->Change this code to not log user-controlled data. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&issues=AYPkb5rB3zqccheRFO5r&open=AYPkb5rB3zqccheRFO5r&pullRequest=12076">SonarCloud</a></p>
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/1887)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -243,13 +261,24 @@
}
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ // Resource resource = resourceMapper.selectById(resourceId);
+ Boolean doesResExist = false;
+ try {
+ doesResExist = storageOperate.exists(fullName);
+ } catch (Exception e) {
+ logger.error("udf resource checking error", fullName);
result.setCode(Status.RESOURCE_NOT_EXIST.getCode());
result.setMsg(Status.RESOURCE_NOT_EXIST.getMsg());
return result;
}
+
+ if (!doesResExist) {
+ logger.error("resource full name {} is not exist", fullName);
Review Comment:
## Logging should not be vulnerable to injection attacks
<!--SONAR_ISSUE_KEY:AYPkb5rB3zqccheRFO5q-->Change this code to not log user-controlled data. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&issues=AYPkb5rB3zqccheRFO5q&open=AYPkb5rB3zqccheRFO5q&pullRequest=12076">SonarCloud</a></p>
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/1888)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1280313527
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276554334
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [117 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996371097
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -285,18 +298,19 @@ export function useTable() {
router.push({ name: 'resource-manage' })
}
- const goBread = (fullName: string) => {
- const { id } = variables
+ const goBread = (fileName: string) => {
+ const id = -1
Review Comment:
ya, the "id" is deprecated. Will remove it in another pr along with all other "ids". Right now they are labeled as Deprecated in the backend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1278508090
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [111 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [29.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1258476950
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1278613413
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [109 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [29.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1274066606
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1279871044
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [31.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1004552114
##########
dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml:
##########
@@ -191,6 +191,22 @@
WHERE td.resource_ids is not null and td.resource_ids != '' and td.user_id = #{userId}
</select>
+ <select id="queryResourcesInUseWithinResourceList" resultType="java.util.HashMap">
+ SELECT distinct pd.code,td.resource_ids
+ FROM t_ds_process_task_relation ptr
+ join t_ds_process_definition pd
+ on ptr.process_definition_code=pd.code and ptr.process_definition_version = pd.version
+ and ptr.project_code=pd.project_code and pd.release_state = 1
+ join t_ds_task_definition td
+ on (ptr.pre_task_code=td.code and ptr.pre_task_version=td.version)
+ or (ptr.post_task_code=td.code and ptr.post_task_version=td.version)
+ WHERE 1 = 1
+ and
+ <foreach item="item" index="index" collection="resourcesArray" separator=" OR " >
+ td.resource_ids LIKE '%${item}%' and td.resource_ids != ''
+ </foreach>
+ </select>
Review Comment:
Right. The behaviour of checking what resources are currently used by tasks were modified and this sql is not in use anymore, therefore I delete it.
> This sql is complex, please split it.
> Avoid using `or` query, it will miss the index.
Right. The behaviour of checking what resources are currently used by tasks were modified and this sql is not in use anymore, therefore I delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1005130559
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/HadoopUtils.java:
##########
@@ -749,4 +899,67 @@ public void deleteTenant(String tenantCode) throws Exception {
public ResUploadType returnStorageType() {
return ResUploadType.HDFS;
}
+
+ @Override
+ public List<StorageEntity> listFilesStatusRecursively(String path, String defaultPath, String tenantCode,
+ ResourceType type) {
+ List<StorageEntity> storageEntityList = new ArrayList<>();
+
+ LinkedList<StorageEntity> foldersToFetch = new LinkedList<>();
+
+ do {
+ String pathToExplore = "";
+ if (foldersToFetch.size() == 0) {
+ pathToExplore = path;
+ } else {
+ pathToExplore = foldersToFetch.pop().getFullName();
+ }
+
+ try {
+ List<StorageEntity> tempList = listFilesStatus(pathToExplore, defaultPath, tenantCode, type);
+
+ for (StorageEntity temp : tempList) {
+ if (temp.isDirectory()) {
+ foldersToFetch.add(temp);
+ }
+ }
+
+ storageEntityList.addAll(tempList);
+ } catch (FileNotFoundException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
+ // return the resources fetched before error occurs.
+ return storageEntityList;
+ } catch (IOException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
Review Comment:
```suggestion
logger.error("Resource path: {}", pathToExplore, e);
```
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -384,4 +433,197 @@ private void deleteDirectory(String directoryName) {
public ResUploadType returnStorageType() {
return ResUploadType.S3;
}
+
+ @Override
+ public List<StorageEntity> listFilesStatusRecursively(String path, String defaultPath, String tenantCode,
+ ResourceType type) {
+ List<StorageEntity> storageEntityList = new ArrayList<>();
+
+ LinkedList<StorageEntity> foldersToFetch = new LinkedList<>();
+
+ do {
+ String pathToExplore = "";
+ if (foldersToFetch.size() == 0) {
+ pathToExplore = path;
+ } else {
+ pathToExplore = foldersToFetch.pop().getFullName();
+ }
+
+ try {
+ List<StorageEntity> tempList = listFilesStatus(pathToExplore, defaultPath, tenantCode, type);
+
+ for (StorageEntity temp : tempList) {
+ if (temp.isDirectory()) {
+ foldersToFetch.add(temp);
+ }
+ }
+
+ storageEntityList.addAll(tempList);
+ } catch (AmazonServiceException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
Review Comment:
```suggestion
logger.error("Resource path: {}", pathToExplore, e);
```
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -171,6 +183,26 @@ public String getResourceFileName(String tenantCode, String fileName) {
return String.format(FORMAT_S_S, getS3ResDir(tenantCode), fileName);
}
+ @Override
+ public String getResourceFileName(String fullName) {
Review Comment:
It seems the same method `getResourceFileName` in `HadoopUtils`?
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -206,21 +238,38 @@ public void download(String tenantCode, String srcFilePath, String dstFilePath,
}
@Override
- public boolean exists(String tenantCode, String fileName) throws IOException {
- return s3Client.doesObjectExist(BUCKET_NAME, fileName);
+ public boolean exists(String fullName) throws IOException {
+ return s3Client.doesObjectExist(BUCKET_NAME, fullName);
}
@Override
- public boolean delete(String tenantCode, String filePath, boolean recursive) throws IOException {
+ public boolean delete(String fullName, boolean recursive) throws IOException {
try {
- s3Client.deleteObject(BUCKET_NAME, filePath);
+ s3Client.deleteObject(BUCKET_NAME, fullName);
return true;
} catch (AmazonServiceException e) {
- logger.error("delete the object error,the resource path is {}", filePath);
+ logger.error("delete the object error,the resource path is {}", fullName);
return false;
}
}
+ @Override
+ public boolean delete(String fullName, List<String> childrenPathList, boolean recursive) throws IOException {
+ // append the resource fullName to the list for deletion.
+ childrenPathList.add(fullName);
+
+ DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(BUCKET_NAME)
+ .withKeys(childrenPathList.stream().toArray(String[]::new));
+ try {
+ s3Client.deleteObjects(deleteObjectsRequest);
+ } catch (AmazonServiceException e) {
+ System.out.println(e.getMessage());
Review Comment:
logger.error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1291369966
Is it need to update doc? Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1278615275
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [109 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [29.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1278506798
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [111 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [29.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996289630
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -355,7 +326,13 @@ private void updateParentResourceSize(Resource resource, long size) {
* @return true if resource exists
*/
private boolean checkResourceExists(String fullName, int type) {
- Boolean existResource = resourcesMapper.existResource(fullName, type);
+ // Boolean existResource = resourcesMapper.existResource(fullName, type);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e) {
+ logger.error("error occurred when checking resource: " + fullName);
Review Comment:
```suggestion
logger.error("error occurred when checking resource: " + fullName, e);
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -1808,4 +1929,44 @@ private List<Resource> queryResourceList(Integer userId, int perm) {
private AuthorizationType checkResourceType(ResourceType type) {
return type.equals(ResourceType.FILE) ? AuthorizationType.RESOURCE_FILE_ID : AuthorizationType.UDF_FILE;
}
+
+ /**
+ * check permission by comparing login user's tenantCode with tenantCode in the request
+ *
+ * @param loginUser user who currently logs in
+ * @param resTenantCode tenantCode in the request field "resTenantCode", can be different from the login user in the case of admin users.
+ * @param result result Object containing different cases
+ * @return tenantCode
+ */
+ private String getTenantCodeIfuserValid(User loginUser, String resTenantCode, Result<Object> result) {
Review Comment:
Please don't use the `Result<Object> result` as input param, we should avoid to change the input param.
You can throw the exception like `throw new ServiceException(Status.USER_NOT_EXIST)` when some check error.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -272,46 +254,34 @@ public Result<Object> createResource(User loginUser,
}
// check resource name exists
- String fullName = getFullName(currentDir, name);
- if (checkResourceExists(fullName, type.ordinal())) {
- logger.warn("Resource exists, can not create again, fullName:{}.", RegexUtils.escapeNRT(name));
- putMsg(result, Status.RESOURCE_EXIST);
- return result;
- }
- if (fullName.length() > Constants.RESOURCE_FULL_NAME_MAX_LENGTH) {
- logger.warn("Resource file's name is longer than max full name length, fullName:{}, fullNameSize:{}, maxFullNameSize:{}",
- RegexUtils.escapeNRT(name), fullName.length(), Constants.RESOURCE_FULL_NAME_MAX_LENGTH);
- putMsg(result, Status.RESOURCE_FULL_NAME_TOO_LONG_ERROR);
- return result;
- }
-
- Date now = new Date();
- Resource resource = new Resource(pid, name, fullName, false, desc, file.getOriginalFilename(),
- loginUser.getId(), type, file.getSize(), now, now);
+ String userResRootPath = ResourceType.UDF.equals(type) ? storageOperate.getUdfDir(tenantCode)
+ : storageOperate.getResDir(tenantCode);
+ String currDirNFileName = !currentDir.contains(userResRootPath) ? userResRootPath + name : currentDir + name;
try {
- resourcesMapper.insert(resource);
- updateParentResourceSize(resource, resource.getSize());
- putMsg(result, Status.SUCCESS);
- permissionPostHandle(resource.getType(), loginUser, resource.getId());
- Map<String, Object> resultMap = new HashMap<>();
- for (Map.Entry<Object, Object> entry : new BeanMap(resource).entrySet()) {
- if (!"class".equalsIgnoreCase(entry.getKey().toString())) {
- resultMap.put(entry.getKey().toString(), entry.getValue());
- }
+ if (checkResourceExists(currDirNFileName, type.ordinal())) {
+ logger.error("resource {} has exist, can't recreate", RegexUtils.escapeNRT(name));
+ putMsg(result, Status.RESOURCE_EXIST);
+ return result;
}
- result.setData(resultMap);
} catch (Exception e) {
- logger.warn("Resource exists, can not create again, fullName:{}.", fullName, e);
throw new ServiceException("resource already exists, can't recreate");
}
+ if (currDirNFileName.length() > Constants.RESOURCE_FULL_NAME_MAX_LENGTH) {
+ logger.warn(
Review Comment:
```suggestion
logger.error(
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -506,63 +441,90 @@ public Result<Object> updateResource(User loginUser,
resource.setSize(file.getSize());
}
- try {
- resourcesMapper.updateById(resource);
- if (resource.isDirectory()) {
- List<Integer> childrenResource = listAllChildren(resource, false);
- if (CollectionUtils.isNotEmpty(childrenResource)) {
- String matcherFullName = Matcher.quoteReplacement(fullName);
- List<Resource> childResourceList;
- Integer[] childResIdArray = childrenResource.toArray(new Integer[childrenResource.size()]);
- List<Resource> resourceList = resourcesMapper.listResourceByIds(childResIdArray);
- childResourceList = resourceList.stream().map(t -> {
- t.setFullName(t.getFullName().replaceFirst(originFullName, matcherFullName));
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- resourcesMapper.batchUpdateResource(childResourceList);
-
- if (ResourceType.UDF.equals(resource.getType())) {
- List<UdfFunc> udfFuncs = udfFunctionMapper.listUdfByResourceId(childResIdArray);
- if (CollectionUtils.isNotEmpty(udfFuncs)) {
- udfFuncs = udfFuncs.stream().map(t -> {
- t.setResourceName(t.getResourceName().replaceFirst(originFullName, matcherFullName));
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- udfFunctionMapper.batchUpdateUdfFunc(udfFuncs);
- }
- }
- }
- } else if (ResourceType.UDF.equals(resource.getType())) {
- List<UdfFunc> udfFuncs = udfFunctionMapper.listUdfByResourceId(new Integer[]{resourceId});
- if (CollectionUtils.isNotEmpty(udfFuncs)) {
- udfFuncs = udfFuncs.stream().map(t -> {
- t.setResourceName(fullName);
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- udfFunctionMapper.batchUpdateUdfFunc(udfFuncs);
- }
+ // if name unchanged, return directly without moving on HDFS
+ if (originResourceName.equals(name) && file == null) {
+ return result;
+ }
+ List<ResourcesTask> existResourcesList;
+ if (resource.isDirectory()) {
+ existResourcesList = resourceTaskMapper.selectSubfoldersFullNames(originFullName + FOLDER_SEPARATOR);
+ } else {
+ existResourcesList = resourceTaskMapper.selectByMap(
+ Collections.singletonMap("full_name", originFullName));
+ }
+
+ if (existResourcesList.size() > 0 && !fullName.equals(originFullName)) {
+ // check if any related task is online. If it is, it can not be updated.
+ for (ResourcesTask existResource : existResourcesList) {
+ int taskId = existResource.getTaskId();
+ if (processService.isTaskOnline(taskDefinitionMapper.selectById(taskId).getCode())) {
+ logger.error("can't be updated,because it is used of process definition that's online");
+ logger.error("resource task relation id:{} is used of task code {}", existResource.getId(),
+ taskDefinitionMapper.selectById(taskId).getCode());
+ putMsg(result, Status.RESOURCE_IS_USED);
+ return result;
+ }
}
- putMsg(result, Status.SUCCESS);
- Map<String, Object> resultMap = new HashMap<>();
- for (Map.Entry<Object, Object> entry : new BeanMap(resource).entrySet()) {
- if (!Constants.CLASS.equalsIgnoreCase(entry.getKey().toString())) {
- resultMap.put(entry.getKey().toString(), entry.getValue());
+ for (ResourcesTask existResource : existResourcesList) {
+ int taskId = existResource.getTaskId();
+ long taskCode = taskDefinitionMapper.selectById(taskId).getCode();
+
+ List<ProcessTaskRelation> processTaskRelation = processTaskRelationMapper.selectByMap(
+ Collections.singletonMap("post_task_code", taskCode));
+ if (processTaskRelation.size() > 0) {
+ long processDefinitionCode = processTaskRelation.get(0).getProcessDefinitionCode();
+ int processDefinitionVersion = processTaskRelation.get(0).getProcessDefinitionVersion();
+ List<ProcessTaskRelation> taskRelationList = processTaskRelationMapper.queryByProcessCode(
+ processTaskRelation.get(0).getProjectCode(),
+ processDefinitionCode);
+
+ List<TaskDefinition> taskDefinitionLogList = new ArrayList<>();
+
+ if (taskRelationList.size() > 0) {
+ ProcessDefinitionLog processDefinition =
+ processDefinitionLogMapper.queryByDefinitionCodeAndVersion(
+ processDefinitionCode, processDefinitionVersion);
+ for (ProcessTaskRelation taskRelation : taskRelationList) {
+ long taskCodeInProcess = taskRelation.getPostTaskCode();
+ TaskDefinition taskDefinition = taskDefinitionMapper.queryByCode(taskCodeInProcess);
+ if (taskCodeInProcess == taskCode) {
+ // originFullName is a prefix if isDirectory is true
+ taskDefinition.setTaskParams(RemoveResourceFromResourceList(originFullName,
+ taskDefinition.getTaskParams(),
+ resource.isDirectory()));
+ // if isDirectory is true, fullName is the new prefix. we replace old prefix
+ // of resource fullname with the new prefix.
+ // if isDirectory is false, fullName is the new path.
+ taskDefinition.setTaskParams(AddResourceToResourceList(originFullName,
+ fullName,
+ existResource.getFullName(),
+ taskDefinition.getTaskParams(),
+ resource.isDirectory()));
Review Comment:
Why call `taskDefinition.setTaskParams` twice?
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -374,38 +351,38 @@ private boolean checkResourceExists(String fullName, int type) {
@Transactional
public Result<Object> updateResource(User loginUser,
int resourceId,
+ String resourceFullName,
+ String resTenantCode,
String name,
String desc,
ResourceType type,
MultipartFile file) {
Result<Object> result = new Result<>();
- String funcPermissionKey = type.equals(ResourceType.FILE) ? ApiFuncIdentificationConstant.FILE_UPDATE
- : ApiFuncIdentificationConstant.UDF_UPDATE;
- boolean canOperatorPermissions =
- canOperatorPermissions(loginUser, new Object[]{resourceId}, checkResourceType(type), funcPermissionKey);
- if (!canOperatorPermissions) {
- putMsg(result, Status.NO_CURRENT_OPERATING_PERMISSION);
- return result;
- }
+
result = checkResourceUploadStartupState();
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
return result;
}
- Resource resource = resourcesMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
- putMsg(result, Status.RESOURCE_NOT_EXIST);
+ String tenantCode = getTenantCodeIfuserValid(loginUser, resTenantCode, result);
+ if (tenantCode == null) {
return result;
}
- if (checkDescriptionLength(desc)) {
- logger.warn("Parameter description is too long.");
- putMsg(result, Status.DESCRIPTION_TOO_LONG_ERROR);
- return result;
+
+ String defaultPath = storageOperate.getResDir(tenantCode);
+
+ StorageEntity resource;
+ try {
+ resource = storageOperate.getFileStatus(resourceFullName, defaultPath, resTenantCode, type);
+ } catch (Exception e) {
+ logger.error(e.getMessage() + " Resource path: {}", resourceFullName, e);
Review Comment:
```suggestion
logger.error("Get file status fail, resource path: {}", resourceFullName, e);
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -374,38 +351,38 @@ private boolean checkResourceExists(String fullName, int type) {
@Transactional
public Result<Object> updateResource(User loginUser,
int resourceId,
+ String resourceFullName,
+ String resTenantCode,
String name,
String desc,
ResourceType type,
MultipartFile file) {
Result<Object> result = new Result<>();
- String funcPermissionKey = type.equals(ResourceType.FILE) ? ApiFuncIdentificationConstant.FILE_UPDATE
- : ApiFuncIdentificationConstant.UDF_UPDATE;
- boolean canOperatorPermissions =
- canOperatorPermissions(loginUser, new Object[]{resourceId}, checkResourceType(type), funcPermissionKey);
- if (!canOperatorPermissions) {
- putMsg(result, Status.NO_CURRENT_OPERATING_PERMISSION);
- return result;
- }
+
result = checkResourceUploadStartupState();
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
return result;
}
- Resource resource = resourcesMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
- putMsg(result, Status.RESOURCE_NOT_EXIST);
+ String tenantCode = getTenantCodeIfuserValid(loginUser, resTenantCode, result);
+ if (tenantCode == null) {
return result;
}
- if (checkDescriptionLength(desc)) {
- logger.warn("Parameter description is too long.");
- putMsg(result, Status.DESCRIPTION_TOO_LONG_ERROR);
- return result;
+
+ String defaultPath = storageOperate.getResDir(tenantCode);
+
+ StorageEntity resource;
+ try {
+ resource = storageOperate.getFileStatus(resourceFullName, defaultPath, resTenantCode, type);
+ } catch (Exception e) {
+ logger.error(e.getMessage() + " Resource path: {}", resourceFullName, e);
+ putMsg(result, Status.RESOURCE_NOT_EXIST);
+ throw new ServiceException(String.format(e.getMessage() + " Resource path: %s", resourceFullName));
Review Comment:
```suggestion
throw new ServiceException(String.format ("Get file status fail, resource path: %s", resourceFullName));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] tang350476527 commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
tang350476527 commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1270280582
谢谢您的来信，您的邮件已收到！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1290991806
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [88 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1291523681
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [87 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1289988811
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [90 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-code-scanning[bot] commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1002625283
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/HadoopUtils.java:
##########
@@ -749,4 +903,67 @@ public void deleteTenant(String tenantCode) throws Exception {
public ResUploadType returnStorageType() {
return ResUploadType.HDFS;
}
+
+ @Override
+ public List<StorageEntity> listFilesStatusRecursively(String path, String defaultPath, String tenantCode,
+ ResourceType type) {
+ List<StorageEntity> storageEntityList = new ArrayList<>();
+
+ LinkedList<StorageEntity> foldersToFetch = new LinkedList<>();
+
+ do {
+ String pathToExplore = "";
+ if (foldersToFetch.size() == 0) {
+ pathToExplore = path;
+ } else {
+ pathToExplore = foldersToFetch.pop().getFullName();
+ }
+
+ try {
+ List<StorageEntity> tempList = listFilesStatus(pathToExplore, defaultPath, tenantCode, type);
+
+ for (StorageEntity temp : tempList) {
+ if (temp.isDirectory()) {
+ foldersToFetch.add(temp);
+ }
+ }
+
+ storageEntityList.addAll(tempList);
+ } catch (FileNotFoundException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
This log entry depends on a [user-provided value](2).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/769)
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -206,21 +237,38 @@ public void download(String tenantCode, String srcFilePath, String dstFilePath,
}
@Override
- public boolean exists(String tenantCode, String fileName) throws IOException {
- return s3Client.doesObjectExist(BUCKET_NAME, fileName);
+ public boolean exists(String fullName) throws IOException {
+ return s3Client.doesObjectExist(BUCKET_NAME, fullName);
}
@Override
- public boolean delete(String tenantCode, String filePath, boolean recursive) throws IOException {
+ public boolean delete(String fullName, boolean recursive) throws IOException {
try {
- s3Client.deleteObject(BUCKET_NAME, filePath);
+ s3Client.deleteObject(BUCKET_NAME, fullName);
return true;
} catch (AmazonServiceException e) {
- logger.error("delete the object error,the resource path is {}", filePath);
+ logger.error("delete the object error,the resource path is {}", fullName);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
This log entry depends on a [user-provided value](2).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/772)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -251,13 +261,23 @@
}
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean doesResExist = false;
+ try {
+ doesResExist = storageOperate.exists(fullName);
+ } catch (Exception e) {
+ logger.error("udf resource checking error", fullName);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/758)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -251,13 +261,23 @@
}
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean doesResExist = false;
+ try {
+ doesResExist = storageOperate.exists(fullName);
+ } catch (Exception e) {
+ logger.error("udf resource checking error", fullName);
result.setCode(Status.RESOURCE_NOT_EXIST.getCode());
result.setMsg(Status.RESOURCE_NOT_EXIST.getMsg());
return result;
}
+
+ if (!doesResExist) {
+ logger.error("resource full name {} is not exist", fullName);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/759)
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -384,4 +432,197 @@
public ResUploadType returnStorageType() {
return ResUploadType.S3;
}
+
+ @Override
+ public List<StorageEntity> listFilesStatusRecursively(String path, String defaultPath, String tenantCode,
+ ResourceType type) {
+ List<StorageEntity> storageEntityList = new ArrayList<>();
+
+ LinkedList<StorageEntity> foldersToFetch = new LinkedList<>();
+
+ do {
+ String pathToExplore = "";
+ if (foldersToFetch.size() == 0) {
+ pathToExplore = path;
+ } else {
+ pathToExplore = foldersToFetch.pop().getFullName();
+ }
+
+ try {
+ List<StorageEntity> tempList = listFilesStatus(pathToExplore, defaultPath, tenantCode, type);
+
+ for (StorageEntity temp : tempList) {
+ if (temp.isDirectory()) {
+ foldersToFetch.add(temp);
+ }
+ }
+
+ storageEntityList.addAll(tempList);
+ } catch (AmazonServiceException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
This log entry depends on a [user-provided value](2).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/775)
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/HadoopUtils.java:
##########
@@ -749,4 +903,67 @@
public ResUploadType returnStorageType() {
return ResUploadType.HDFS;
}
+
+ @Override
+ public List<StorageEntity> listFilesStatusRecursively(String path, String defaultPath, String tenantCode,
+ ResourceType type) {
+ List<StorageEntity> storageEntityList = new ArrayList<>();
+
+ LinkedList<StorageEntity> foldersToFetch = new LinkedList<>();
+
+ do {
+ String pathToExplore = "";
+ if (foldersToFetch.size() == 0) {
+ pathToExplore = path;
+ } else {
+ pathToExplore = foldersToFetch.pop().getFullName();
+ }
+
+ try {
+ List<StorageEntity> tempList = listFilesStatus(pathToExplore, defaultPath, tenantCode, type);
+
+ for (StorageEntity temp : tempList) {
+ if (temp.isDirectory()) {
+ foldersToFetch.add(temp);
+ }
+ }
+
+ storageEntityList.addAll(tempList);
+ } catch (FileNotFoundException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
+ // return the resources fetched before error occurs.
+ return storageEntityList;
+ } catch (IOException e) {
+ logger.error(e.getMessage() + "Resource path:" + pathToExplore, e);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
This log entry depends on a [user-provided value](2).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/770)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -117,9 +120,15 @@
return result;
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e) {
+ logger.error("AmazonServiceException when checking resource: " + fullName);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/755)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -117,9 +120,15 @@
return result;
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e) {
+ logger.error("AmazonServiceException when checking resource: " + fullName);
+ }
+
+ if (!existResource) {
+ logger.error("resource full name {} is not exist", fullName);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/756)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1287992684
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [89 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1279871302
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1279867912
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276722830
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [114 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1257403738
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1004148315
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -171,6 +183,25 @@ public String getResourceFileName(String tenantCode, String fileName) {
return String.format(FORMAT_S_S, getS3ResDir(tenantCode), fileName);
}
+ @Override
+ public String getResourceFileName(String fullName) {
+ // here is a quick fix here to get fileName. We get the resource upload path and
+ // get the index of the first appearance of resource upload path. The index is put
+ // in the start index of the substring function and get the result substring containing
+ // tenantcode and "resource" directory and the fileName.
+ // Then we split the result substring
+ // with "/" and join all elements except the first two elements because they are
+ // tenantCode and "resource" directory.
+ String resourceUploadPath =
+ RESOURCE_UPLOAD_PATH.endsWith("/") ? StringUtils.chop(RESOURCE_UPLOAD_PATH) : RESOURCE_UPLOAD_PATH;
+ // +1 because we want to skip the "/" after resource upload path as well.
+ String pathContainingTenantNResource = fullName.substring(
+ fullName.indexOf(resourceUploadPath)
+ + resourceUploadPath.length() + 1);
+ String[] fileNameArr = pathContainingTenantNResource.split("/");
+ return Joiner.on("/").join(Arrays.stream(fileNameArr).skip(2).collect(Collectors.toList()));
+ }
Review Comment:
```suggestion
public String getResourceFileName(String fullName) {
// here is a quick fix here to get fileName. We get the resource upload path and
// get the index of the first appearance of resource upload path. The index is put
// in the start index of the substring function and get the result substring containing
// tenantcode and "resource" directory and the fileName.
// Then we split the result substring
// with "/" and join all elements except the first two elements because they are
// tenantCode and "resource" directory.
String resourceUploadPath =
RESOURCE_UPLOAD_PATH.endsWith(Constnats.FOLDER_SEPARATOR) ? StringUtils.chop(RESOURCE_UPLOAD_PATH) : RESOURCE_UPLOAD_PATH;
// +1 because we want to skip the "/" after resource upload path as well.
String pathContainingTenantNResource = fullName.substring(
fullName.indexOf(resourceUploadPath)
+ resourceUploadPath.length() + 1);
String[] fileNameArr = pathContainingTenantNResource.split(Constnats.FOLDER_SEPARATOR);
return Joiner.on(Constnats.FOLDER_SEPARATOR).join(Arrays.stream(fileNameArr).skip(2).collect(Collectors.toList()));
}
```
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/HadoopUtils.java:
##########
@@ -290,14 +302,33 @@ public String getUdfDir(String tenantCode) {
*/
@Override
public boolean mkdir(String tenantCode, String hdfsPath) throws IOException {
- return fs.mkdirs(new Path(hdfsPath));
+ return fs.mkdirs(new Path(addFolderSeparatorIfNotExisted(hdfsPath)));
}
@Override
public String getResourceFileName(String tenantCode, String fullName) {
return getHdfsResourceFileName(tenantCode, fullName);
}
+ @Override
+ public String getResourceFileName(String fullName) {
+ // here is a quick fix here to get fileName. We get the resource upload path and
+ // get the index of the first appearance of resource upload path. The index is put
+ // in the start index of the substring function and get the result substring containing
+ // tenantcode and "resource" directory and the fileName.
+ // Then we split the result substring
+ // with "/" and join all elements except the first two elements because they are
+ // tenantCode and "resource" directory.
+ String resourceUploadPath =
+ RESOURCE_UPLOAD_PATH.endsWith("/") ? StringUtils.chop(RESOURCE_UPLOAD_PATH) : RESOURCE_UPLOAD_PATH;
+ // +1 because we want to skip the "/" after resource upload path as well.
+ String pathContainingTenantNResource = fullName.substring(
+ fullName.indexOf(resourceUploadPath)
+ + resourceUploadPath.length() + 1);
+ String[] fileNameArr = pathContainingTenantNResource.split("/");
+ return Joiner.on("/").join(Arrays.stream(fileNameArr).skip(2).collect(Collectors.toList()));
+ }
Review Comment:
replace "/" with `Constnats.FOLDER_SEPARATOR`
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/HadoopUtils.java:
##########
@@ -321,6 +352,11 @@ public void download(String bucketName, String srcHdfsFilePath, String dstFile,
*/
@Override
public boolean copy(String srcPath, String dstPath, boolean deleteSource, boolean overwrite) throws IOException {
+ // if(!exists(dstPath))
+ // {
+ // fs.mkdirs(new Path(dstPath));
+ // logger.info("created the destination path: {}", dstPath);
+ // }
Review Comment:
remove if not use.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Amy0104 commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
Amy0104 commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996919388
##########
dolphinscheduler-ui/src/views/resource/udf/function/components/use-modal.ts:
##########
@@ -84,9 +82,9 @@ export function useModal(
const filterEmptyDirectory = (list: any) => {
for (const item of list) {
if (item.children) {
- if (!/\.jar$/.test(item.name)) {
- item.disabled = true
- }
+// if (!/\.jar$/.test(item.name)) {
+// item.disabled = true
+// }
Review Comment:
Pls remove the useless comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Amy0104 commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
Amy0104 commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996256663
##########
dolphinscheduler-ui/src/views/resource/file/create/use-create.ts:
##########
@@ -27,7 +27,7 @@ export function useCreate(state: any) {
const fileStore = useFileStore()
const handleCreateFile = () => {
- const pid = router.currentRoute.value.params.id || -1
+ const pid = -1
Review Comment:
It seems to need to remove `pid`.
##########
dolphinscheduler-ui/src/views/projects/task/components/node/types.ts:
##########
@@ -91,7 +91,9 @@ interface ISwitchResult {
}
interface ISourceItem {
- id: number
+ id?: number,
+ resourceName: string,
+ res?: string
Review Comment:
what's the meaning of `res`? Would it be better to use a full word？
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -242,13 +240,26 @@ export function useTable() {
variables.loadingRef = true
const { state } = useAsyncState(
queryResourceListPaging({ ...params, type: 'UDF' }).then((res: any) => {
- const breadList =
- variables.id === -1
- ? []
- : (fileStore.getCurrentDir.split('/') as Array<never>)
- breadList.shift()
-
- variables.breadList = breadList
+ if (variables.fileId != ""){
Review Comment:
It better to use strict inequality here.
##########
dolphinscheduler-ui/src/views/resource/file/edit/index.tsx:
##########
@@ -32,21 +32,22 @@ export default defineComponent({
const router = useRouter()
const componentName = route.name
- const fileId = Number(route.params.id)
+ const fileId = String(router.currentRoute.value.query.prefix || "")
Review Comment:
Here `fullName` seems to be better than `fileId`.
##########
dolphinscheduler-ui/src/views/resource/file/edit/use-edit.ts:
##########
@@ -28,25 +28,29 @@ export function useEdit(state: any) {
const { t } = useI18n()
const router: Router = useRouter()
- const getResourceView = (id: number) => {
+ const getResourceView = (fullName: string, tenantCode: string, id: number) => {
const params = {
skipLineNum: 0,
- limit: 3000
+ limit: 3000,
+ fullName: fullName,
+ tenantCode: tenantCode
}
return useAsyncState(viewResource(params, id), {
alias: '',
content: ''
})
}
- const handleUpdateContent = (id: number) => {
+ const handleUpdateContent = (id: string, fullName: string, tenantCode: string) => {
Review Comment:
`id` needs to be removed, it seems to be unused.
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -242,13 +240,26 @@ export function useTable() {
variables.loadingRef = true
const { state } = useAsyncState(
queryResourceListPaging({ ...params, type: 'UDF' }).then((res: any) => {
- const breadList =
- variables.id === -1
- ? []
- : (fileStore.getCurrentDir.split('/') as Array<never>)
- breadList.shift()
-
- variables.breadList = breadList
+ if (variables.fileId != ""){
+ const id = -1
+ queryCurrentResourceById(
+ {
+ id,
+ type: 'UDF',
+ fullName: variables.fileId,
+ tenantCode: variables.tenantCode,
+ },
+ id
+ ).then((res: ResourceFile) => {
+ if (res.fileName) {
+ const breadList = res.fileName.split('/') as Array<never>
Review Comment:
The type of `breadList` can not to be `Array<never>`.
##########
dolphinscheduler-ui/src/views/resource/file/index.tsx:
##########
@@ -57,8 +57,8 @@ export default defineComponent({
name: 'File',
setup() {
const router: Router = useRouter()
- const fileId = ref(Number(router.currentRoute.value.params.id) || -1)
-
+ const fileId = ref(String(router.currentRoute.value.query.prefix || ""))
Review Comment:
Same as above.
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -285,18 +298,19 @@ export function useTable() {
router.push({ name: 'resource-manage' })
}
- const goBread = (fullName: string) => {
- const { id } = variables
+ const goBread = (fileName: string) => {
+ const id = -1
Review Comment:
If `id` is useless, it is better to remove it.
##########
dolphinscheduler-ui/src/views/resource/file/upload/use-upload.ts:
##########
@@ -37,7 +34,7 @@ export function useUpload(state: any) {
if (state.saving) return
state.saving = true
try {
- const pid = router.currentRoute.value.params.id || -1
+ const pid = -1
Review Comment:
It's better to remove the `pid`, if it is useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996844238
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java:
##########
@@ -49,37 +49,7 @@
import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils.CodeGenerateException;
import org.apache.dolphinscheduler.common.utils.DateUtils;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
-import org.apache.dolphinscheduler.dao.entity.Command;
-import org.apache.dolphinscheduler.dao.entity.DagData;
-import org.apache.dolphinscheduler.dao.entity.DataSource;
-import org.apache.dolphinscheduler.dao.entity.DependentProcessDefinition;
-import org.apache.dolphinscheduler.dao.entity.DqComparisonType;
-import org.apache.dolphinscheduler.dao.entity.DqExecuteResult;
-import org.apache.dolphinscheduler.dao.entity.DqRule;
-import org.apache.dolphinscheduler.dao.entity.DqRuleExecuteSql;
-import org.apache.dolphinscheduler.dao.entity.DqRuleInputEntry;
-import org.apache.dolphinscheduler.dao.entity.DqTaskStatisticsValue;
-import org.apache.dolphinscheduler.dao.entity.Environment;
-import org.apache.dolphinscheduler.dao.entity.ErrorCommand;
-import org.apache.dolphinscheduler.dao.entity.K8s;
-import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
-import org.apache.dolphinscheduler.dao.entity.ProcessDefinitionLog;
-import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
-import org.apache.dolphinscheduler.dao.entity.ProcessInstanceMap;
-import org.apache.dolphinscheduler.dao.entity.ProcessTaskRelation;
-import org.apache.dolphinscheduler.dao.entity.ProcessTaskRelationLog;
-import org.apache.dolphinscheduler.dao.entity.Project;
-import org.apache.dolphinscheduler.dao.entity.ProjectUser;
-import org.apache.dolphinscheduler.dao.entity.Resource;
-import org.apache.dolphinscheduler.dao.entity.Schedule;
-import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
-import org.apache.dolphinscheduler.dao.entity.TaskDefinitionLog;
-import org.apache.dolphinscheduler.dao.entity.TaskGroup;
-import org.apache.dolphinscheduler.dao.entity.TaskGroupQueue;
-import org.apache.dolphinscheduler.dao.entity.TaskInstance;
-import org.apache.dolphinscheduler.dao.entity.Tenant;
-import org.apache.dolphinscheduler.dao.entity.UdfFunc;
-import org.apache.dolphinscheduler.dao.entity.User;
+import org.apache.dolphinscheduler.dao.entity.*;
Review Comment:
`wildcard import` is supposed to be detected and blocked by Spotless. Violations should cause CI to fail. Looks like something is not correct with Spotless configurations and it missed to catch this violation. I've created an issue to track this: https://github.com/apache/dolphinscheduler/issues/12401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1253136236
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1257387035
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r990737876
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -265,15 +281,18 @@ export function useTable() {
variables.row = row
}
- const handleDelete = (id: number) => {
+ const handleDelete = (id: number, fullNameObj: {fullName: string, tenantCode: string}) => {
/* after deleting data from the current page, you need to jump forward when the page is empty. */
if (variables.tableData.length === 1 && variables.page > 1) {
variables.page -= 1
}
- deleteResource(id).then(() =>
+ deleteResource(id, fullNameObj).then(() =>
getTableData({
- id: variables.id,
+ id: -1,
+// id: variables.id,
Review Comment:
Same here.
##########
dolphinscheduler-ui/src/views/resource/file/table/use-table.ts:
##########
@@ -86,13 +85,13 @@ export function useTable(renameResource: IRenameFile, updateList: () => void) {
{
title: t('resource.file.file_name'),
...COLUMN_WIDTH_CONFIG['name'],
- key: 'file_name'
- },
- {
- title: t('resource.file.description'),
- ...COLUMN_WIDTH_CONFIG['note'],
- key: 'description'
+ key: 'fullName'
},
+// {
+// title: t('resource.file.description'),
+// ...COLUMN_WIDTH_CONFIG['note'],
+// key: 'description'
+// },
Review Comment:
Same here.
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -117,19 +120,19 @@ export function useTable() {
{
title: t('resource.udf.file_name'),
...COLUMN_WIDTH_CONFIG['name'],
- key: 'fileName'
+ key: 'fullName'
},
{
title: t('resource.udf.file_size'),
key: 'size',
...COLUMN_WIDTH_CONFIG['size'],
render: (row) => bytesToSize(row.size)
},
- {
- title: t('resource.udf.description'),
- key: 'description',
- ...COLUMN_WIDTH_CONFIG['note']
- },
+// {
+// title: t('resource.udf.description'),
+// key: 'description',
+// ...COLUMN_WIDTH_CONFIG['note']
+// },
Review Comment:
Please remove unnessnary comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1280316663
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [32.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r976525897
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -790,6 +790,22 @@ CREATE TABLE `t_ds_resources` (
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_relation_resources_task`;
+CREATE TABLE `t_ds_relation_resources_task` (
+ `id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
+ `full_name` varchar(255) DEFAULT NULL,
+ `type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `t_ds_relation_resources_task_un` (`full_name`,`type`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3
Review Comment:
> I think this table is about resources, not the relation between task and resources. Why not create this table like this, and then we can remove the `resource_ids_new` field in `t_ds_task_definition`
It is a relation between tasks and resources because we store resources into this table only when a task uses them.
Having a `task_id` field in table definitely helps a lot! I will make some changes based on this new field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1253110225
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1253133115
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1289985684
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [90 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1004126885
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -117,9 +120,15 @@ public Result<Object> createUdfFunction(User loginUser,
return result;
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e) {
+ logger.error("AmazonServiceException when checking resource: " + fullName);
Review Comment:
```suggestion
logger.error("Check resource error: {}", fullName, e);
```
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ResourcesTask.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.dolphinscheduler.dao.entity;
+
+import org.apache.dolphinscheduler.spi.enums.ResourceType;
+
+import lombok.Data;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+
+@Data
+@TableName("t_ds_relation_resources_task")
+public class ResourcesTask {
+
+ private String fullName;
+
+ @TableId(value = "id", type = IdType.AUTO)
+ private Integer id;
Review Comment:
```suggestion
@TableId(value = "id", type = IdType.AUTO)
private Integer id;
private String fullName;
```
##########
dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ResourceTaskMapper.xml:
##########
@@ -0,0 +1,58 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
+<mapper namespace="org.apache.dolphinscheduler.dao.mapper.ResourceTaskMapper">
+ <sql id="baseSqlV2">
+ ${alias}.id, ${alias}.full_name, ${alias}.task_id, ${alias}.type
+ </sql>
+ <select id="existResourceByTaskIdNFullName" resultType="java.lang.Integer">
+ select
+ id
+ from t_ds_relation_resources_task
+ where full_name = #{fullName} and task_id = #{taskId} limit 1
Review Comment:
Why limit 1 here? The taskId and fullName is not uniq?
##########
dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml:
##########
@@ -191,6 +191,22 @@
WHERE td.resource_ids is not null and td.resource_ids != '' and td.user_id = #{userId}
</select>
+ <select id="queryResourcesInUseWithinResourceList" resultType="java.util.HashMap">
+ SELECT distinct pd.code,td.resource_ids
+ FROM t_ds_process_task_relation ptr
+ join t_ds_process_definition pd
+ on ptr.process_definition_code=pd.code and ptr.process_definition_version = pd.version
+ and ptr.project_code=pd.project_code and pd.release_state = 1
+ join t_ds_task_definition td
+ on (ptr.pre_task_code=td.code and ptr.pre_task_version=td.version)
+ or (ptr.post_task_code=td.code and ptr.post_task_version=td.version)
+ WHERE 1 = 1
+ and
+ <foreach item="item" index="index" collection="resourcesArray" separator=" OR " >
+ td.resource_ids LIKE '%${item}%' and td.resource_ids != ''
+ </foreach>
+ </select>
Review Comment:
This sql is complex, please split it.
Avoid using `or` query, it will miss the index.
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ResourcesTask.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.dolphinscheduler.dao.entity;
+
+import org.apache.dolphinscheduler.spi.enums.ResourceType;
+
+import lombok.Data;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+
+@Data
+@TableName("t_ds_relation_resources_task")
+public class ResourcesTask {
+
+ private String fullName;
+
+ @TableId(value = "id", type = IdType.AUTO)
+ private Integer id;
+
+ private int taskId;
+
+ private ResourceType type;
+
+ public ResourcesTask(int id, String fullName, int taskId, ResourceType type) {
+ this.id = id;
+ this.fullName = fullName;
+ this.taskId = taskId;
+ this.type = type;
+ }
+
+ public ResourcesTask(int taskId, String fullName, ResourceType type) {
+ this.taskId = taskId;
+ this.fullName = fullName;
+ this.type = type;
+ }
+
+ @Override
+ public String toString() {
Review Comment:
remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276666337
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [117 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276729462
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [114 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276549862
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [117 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1275665867
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996635654
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -790,6 +790,23 @@ CREATE TABLE `t_ds_resources` (
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_relation_resources_task`;
+CREATE TABLE `t_ds_relation_resources_task` (
+ `id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
+ `task_id` int(11) DEFAULT NULL COMMENT 'task id',
+ `full_name` varchar(255) DEFAULT NULL,
+ `type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `t_ds_relation_resources_task_un` (`task_id`, `full_name`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3;
Review Comment:
```suggestion
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -243,13 +261,24 @@ public Result<Object> updateUdfFunc(User loginUser,
}
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ // Resource resource = resourceMapper.selectById(resourceId);
Review Comment:
Please remove unused comment.
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java:
##########
@@ -49,37 +49,7 @@
import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils.CodeGenerateException;
import org.apache.dolphinscheduler.common.utils.DateUtils;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
-import org.apache.dolphinscheduler.dao.entity.Command;
-import org.apache.dolphinscheduler.dao.entity.DagData;
-import org.apache.dolphinscheduler.dao.entity.DataSource;
-import org.apache.dolphinscheduler.dao.entity.DependentProcessDefinition;
-import org.apache.dolphinscheduler.dao.entity.DqComparisonType;
-import org.apache.dolphinscheduler.dao.entity.DqExecuteResult;
-import org.apache.dolphinscheduler.dao.entity.DqRule;
-import org.apache.dolphinscheduler.dao.entity.DqRuleExecuteSql;
-import org.apache.dolphinscheduler.dao.entity.DqRuleInputEntry;
-import org.apache.dolphinscheduler.dao.entity.DqTaskStatisticsValue;
-import org.apache.dolphinscheduler.dao.entity.Environment;
-import org.apache.dolphinscheduler.dao.entity.ErrorCommand;
-import org.apache.dolphinscheduler.dao.entity.K8s;
-import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
-import org.apache.dolphinscheduler.dao.entity.ProcessDefinitionLog;
-import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
-import org.apache.dolphinscheduler.dao.entity.ProcessInstanceMap;
-import org.apache.dolphinscheduler.dao.entity.ProcessTaskRelation;
-import org.apache.dolphinscheduler.dao.entity.ProcessTaskRelationLog;
-import org.apache.dolphinscheduler.dao.entity.Project;
-import org.apache.dolphinscheduler.dao.entity.ProjectUser;
-import org.apache.dolphinscheduler.dao.entity.Resource;
-import org.apache.dolphinscheduler.dao.entity.Schedule;
-import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
-import org.apache.dolphinscheduler.dao.entity.TaskDefinitionLog;
-import org.apache.dolphinscheduler.dao.entity.TaskGroup;
-import org.apache.dolphinscheduler.dao.entity.TaskGroupQueue;
-import org.apache.dolphinscheduler.dao.entity.TaskInstance;
-import org.apache.dolphinscheduler.dao.entity.Tenant;
-import org.apache.dolphinscheduler.dao.entity.UdfFunc;
-import org.apache.dolphinscheduler.dao.entity.User;
+import org.apache.dolphinscheduler.dao.entity.*;
Review Comment:
Avoid import `*`.
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java:
##########
@@ -2523,7 +2500,7 @@ public int saveTaskDefine(User operator, long projectCode, List<TaskDefinitionLo
taskDefinitionLog.setUpdateTime(now);
taskDefinitionLog.setOperateTime(now);
taskDefinitionLog.setOperator(operator.getId());
- taskDefinitionLog.setResourceIds(getResourceIds(taskDefinitionLog));
+// taskDefinitionLog.setResourceIds(getResourceIds(taskDefinitionLog));
Review Comment:
Please remove unused comment.
##########
dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java:
##########
@@ -17,18 +17,63 @@
package org.apache.dolphinscheduler.service.process;
-import com.fasterxml.jackson.databind.JsonNode;
+import static org.apache.dolphinscheduler.common.Constants.*;
Review Comment:
Avoid import `*`
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java:
##########
@@ -557,8 +557,9 @@ private List<String> buildJarSql(List<UdfFuncParameters> udfFuncParameters) {
String prefixPath = defaultFS.startsWith("file://") ? "file://" : defaultFS;
String uploadPath = CommonUtils.getHdfsUdfDir(value.getTenantCode());
String resourceFullName = value.getResourceName();
- resourceFullName = resourceFullName.startsWith("/") ? resourceFullName : String.format("/%s", resourceFullName);
- return String.format("add jar %s%s%s", prefixPath, uploadPath, resourceFullName);
+// resourceFullName = resourceFullName.startsWith("/") ? resourceFullName : String.format("/%s", resourceFullName);
+// return String.format("add jar %s%s%s", prefixPath, uploadPath, resourceFullName);
Review Comment:
Please remove unused comment.
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/BaseTaskProcessor.java:
##########
@@ -622,25 +622,26 @@ public Map<String, String> getResourceFullNames(TaskInstance taskInstance) {
if (CollectionUtils.isNotEmpty(projectResourceFiles)) {
// filter the resources that the resource id equals 0
- Set<ResourceInfo> oldVersionResources =
- projectResourceFiles.stream().filter(t -> t.getId() == null).collect(Collectors.toSet());
- if (CollectionUtils.isNotEmpty(oldVersionResources)) {
- oldVersionResources.forEach(t -> resourcesMap.put(t.getRes(),
- processService.queryTenantCodeByResName(t.getRes(), ResourceType.FILE)));
- }
-
- // get the resource id in order to get the resource names in batch
- Stream<Integer> resourceIdStream = projectResourceFiles.stream().map(ResourceInfo::getId);
- Set<Integer> resourceIdsSet = resourceIdStream.collect(Collectors.toSet());
-
- if (CollectionUtils.isNotEmpty(resourceIdsSet)) {
- Integer[] resourceIds = resourceIdsSet.toArray(new Integer[resourceIdsSet.size()]);
-
- List<Resource> resources = processService.listResourceByIds(resourceIds);
- resources.forEach(t -> resourcesMap.put(t.getFullName(),
- processService.queryTenantCodeByResName(t.getFullName(), ResourceType.FILE)));
- }
- }
+ // TODO: Modify this part to accomodate oldversionresources in the future.
+// Set<ResourceInfo> oldVersionResources =
+// projectResourceFiles.stream().filter(t -> t.getId() == null).collect(Collectors.toSet());
+// if (CollectionUtils.isNotEmpty(oldVersionResources)) {
+// oldVersionResources.forEach(t -> resourcesMap.put(t.getRes(),
+// processService.queryTenantCodeByResName(t.getRes(), ResourceType.FILE)));
+// }
+//
+// // get the resource id in order to get the resource names in batch
+// Stream<Integer> resourceIdStream = projectResourceFiles.stream().map(ResourceInfo::getId);
+// Set<Integer> resourceIdsSet = resourceIdStream.collect(Collectors.toSet());
+//
+// if (CollectionUtils.isNotEmpty(resourceIdsSet)) {
+// Integer[] resourceIds = resourceIdsSet.toArray(new Integer[resourceIdsSet.size()]);
+//
+// List<Resource> resources = processService.listResourceByIds(resourceIds);
+// resources.forEach(t -> resourcesMap.put(t.getFullName(),
+// processService.queryTenantCodeByResName(t.getFullName(), ResourceType.FILE)));
+// }
Review Comment:
Please remove unused comment.
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql:
##########
@@ -792,6 +792,24 @@ CREATE TABLE t_ds_resources
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS t_ds_relation_resources_task CASCADE;
+CREATE TABLE t_ds_relation_resources_task
+(
+ id int(11) NOT NULL AUTO_INCREMENT,
+ task_id int(11) DEFAULT NULL,
+ full_name varchar(255) DEFAULT NULL,
+ type tinyint(4) DEFAULT NULL,
+ PRIMARY KEY (id),
+ UNIQUE KEY t_ds_relation_resources_task_un (task_id, full_name)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3;
Review Comment:
```suggestion
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET= utf8;
```
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/StorageEntity.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.dolphinscheduler.service.storage;
+
+import org.apache.dolphinscheduler.spi.enums.ResourceType;
+
+import java.util.Date;
+
+// Could be put into org.apache.dolphinscheduler.common.model module?
+// MayBe spi
+public class StorageEntity {
Review Comment:
It's better to use lombok `@Data` instead of get/set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1278501932
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/12076?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#12076](https://codecov.io/gh/apache/dolphinscheduler/pull/12076?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1246b88) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/b953cdc559a49e47ef64cb91f5c063d39a5e54bb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b953cdc) will **decrease** coverage by `0.67%`.
> The diff coverage is `14.14%`.
```diff
@@ Coverage Diff @@
## dev #12076 +/- ##
============================================
- Coverage 38.92% 38.24% -0.68%
+ Complexity 4173 4146 -27
============================================
Files 1040 1042 +2
Lines 38797 39189 +392
Branches 4462 4477 +15
============================================
- Hits 15100 14988 -112
- Misses 21931 22442 +511
+ Partials 1766 1759 -7
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/12076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...scheduler/api/dto/resources/ResourceComponent.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2R0by9yZXNvdXJjZXMvUmVzb3VyY2VDb21wb25lbnQuamF2YQ==) | `25.00% <0.00%> (-6.25%)` | :arrow_down: |
| [...heduler/api/service/impl/ResourcesServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9SZXNvdXJjZXNTZXJ2aWNlSW1wbC5qYXZh) | `39.93% <ø> (-12.21%)` | :arrow_down: |
| [...inscheduler/api/service/impl/UsersServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Vc2Vyc1NlcnZpY2VJbXBsLmphdmE=) | `70.79% <0.00%> (+0.18%)` | :arrow_up: |
| [...che/dolphinscheduler/dao/entity/ResourcesTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9SZXNvdXJjZXNUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...uler/dao/utils/ResourceProcessDefinitionUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL3V0aWxzL1Jlc291cmNlUHJvY2Vzc0RlZmluaXRpb25VdGlscy5qYXZh) | `39.39% <0.00%> (-37.08%)` | :arrow_down: |
| [...e/dolphinscheduler/server/master/MasterServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9NYXN0ZXJTZXJ2ZXIuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...r/server/master/runner/task/BaseTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9CYXNlVGFza1Byb2Nlc3Nvci5qYXZh) | `16.47% <0.00%> (+0.66%)` | :arrow_up: |
| [...olphinscheduler/service/storage/StorageEntity.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2Uvc3RvcmFnZS9TdG9yYWdlRW50aXR5LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...lphinscheduler/service/storage/StorageOperate.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2Uvc3RvcmFnZS9TdG9yYWdlT3BlcmF0ZS5qYXZh) | `100.00% <ø> (ø)` | |
| [...hinscheduler/service/storage/impl/OssOperator.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2Uvc3RvcmFnZS9pbXBsL09zc09wZXJhdG9yLmphdmE=) | `33.06% <0.00%> (-1.11%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/dolphinscheduler/pull/12076/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1290989146
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [88 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1005196240
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -206,21 +238,38 @@ public void download(String tenantCode, String srcFilePath, String dstFilePath,
}
@Override
- public boolean exists(String tenantCode, String fileName) throws IOException {
- return s3Client.doesObjectExist(BUCKET_NAME, fileName);
+ public boolean exists(String fullName) throws IOException {
+ return s3Client.doesObjectExist(BUCKET_NAME, fullName);
}
@Override
- public boolean delete(String tenantCode, String filePath, boolean recursive) throws IOException {
+ public boolean delete(String fullName, boolean recursive) throws IOException {
try {
- s3Client.deleteObject(BUCKET_NAME, filePath);
+ s3Client.deleteObject(BUCKET_NAME, fullName);
return true;
} catch (AmazonServiceException e) {
- logger.error("delete the object error,the resource path is {}", filePath);
+ logger.error("delete the object error,the resource path is {}", fullName);
return false;
}
}
+ @Override
+ public boolean delete(String fullName, List<String> childrenPathList, boolean recursive) throws IOException {
+ // append the resource fullName to the list for deletion.
+ childrenPathList.add(fullName);
+
+ DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(BUCKET_NAME)
+ .withKeys(childrenPathList.stream().toArray(String[]::new));
+ try {
+ s3Client.deleteObjects(deleteObjectsRequest);
+ } catch (AmazonServiceException e) {
+ logger.error(e.getMessage());
Review Comment:
```suggestion
logger.error("delete objects error", e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1274043513
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1259686170
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
mergeable[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1270279787
:warning: This PR do not change database DDL synchronize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r979512839
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -790,6 +790,22 @@ CREATE TABLE `t_ds_resources` (
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_relation_resources_task`;
+CREATE TABLE `t_ds_relation_resources_task` (
+ `id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
+ `full_name` varchar(255) DEFAULT NULL,
+ `type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `t_ds_relation_resources_task_un` (`full_name`,`type`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3
Review Comment:
@caishunfeng @SbloodyS In the latest commit, I added the `task_id` field. While testing it, I found there was no logic in updating task definition-related tables in the repo. For example, In the current implementation, the modification to resources (rename, delete) used by a task won't update the task definition log, which it should. Therefore, at the end I used `processDefinitionService.updateProcessDefinition()` to implement the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r979512839
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -790,6 +790,22 @@ CREATE TABLE `t_ds_resources` (
-- Records of t_ds_resources
-- ----------------------------
+-- ----------------------------
+-- Table structure for t_ds_relation_resources_task
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_relation_resources_task`;
+CREATE TABLE `t_ds_relation_resources_task` (
+ `id` int NOT NULL AUTO_INCREMENT COMMENT 'key',
+ `full_name` varchar(255) DEFAULT NULL,
+ `type` tinyint DEFAULT NULL COMMENT 'resource type,0:FILE,1:UDF',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `t_ds_relation_resources_task_un` (`full_name`,`type`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb3
Review Comment:
@caishunfeng In the latest commit, I added the `task_id` field. While testing it, I found there was no logic in updating task definition-related tables in the repo. For example, In the current implementation, the modification to resources (rename, delete) used by a task won't update the task definition log, which it should. Therefore, at the end I used `processDefinitionService.updateProcessDefinition()` to implement the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-code-scanning[bot] commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r993760695
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UdfFuncServiceImpl.java:
##########
@@ -112,9 +120,15 @@
return result;
}
- Resource resource = resourceMapper.selectById(resourceId);
- if (resource == null) {
- logger.error("Resource does not exist, resourceId:{}.", resourceId);
+ Boolean existResource = false;
+ try {
+ existResource = storageOperate.exists(fullName);
+ } catch (IOException e){
+ logger.error("AmazonServiceException when checking resource: " + fullName);
+ }
+
+ if (!existResource){
+ logger.error("resourceId {} is not exist", fullName);
Review Comment:
## Logging should not be vulnerable to injection attacks
<!--SONAR_ISSUE_KEY:AYPNYBjwUAsZPIwYbSMr-->Change this code to not log user-controlled data. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&issues=AYPNYBjwUAsZPIwYbSMr&open=AYPNYBjwUAsZPIwYbSMr&pullRequest=12076">SonarCloud</a></p>
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/1786)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1276664249
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [117 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [28.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [6.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1279866649
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [11 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [105 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [31.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [10.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996371097
##########
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts:
##########
@@ -285,18 +298,19 @@ export function useTable() {
router.push({ name: 'resource-manage' })
}
- const goBread = (fullName: string) => {
- const { id } = variables
+ const goBread = (fileName: string) => {
+ const id = -1
Review Comment:
ya, the "id" is deprecated. Will remove it in another pr with new routes along with all other "ids". Right now they are labelled as Deprecated in the backend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r996368313
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java:
##########
@@ -506,63 +441,90 @@ public Result<Object> updateResource(User loginUser,
resource.setSize(file.getSize());
}
- try {
- resourcesMapper.updateById(resource);
- if (resource.isDirectory()) {
- List<Integer> childrenResource = listAllChildren(resource, false);
- if (CollectionUtils.isNotEmpty(childrenResource)) {
- String matcherFullName = Matcher.quoteReplacement(fullName);
- List<Resource> childResourceList;
- Integer[] childResIdArray = childrenResource.toArray(new Integer[childrenResource.size()]);
- List<Resource> resourceList = resourcesMapper.listResourceByIds(childResIdArray);
- childResourceList = resourceList.stream().map(t -> {
- t.setFullName(t.getFullName().replaceFirst(originFullName, matcherFullName));
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- resourcesMapper.batchUpdateResource(childResourceList);
-
- if (ResourceType.UDF.equals(resource.getType())) {
- List<UdfFunc> udfFuncs = udfFunctionMapper.listUdfByResourceId(childResIdArray);
- if (CollectionUtils.isNotEmpty(udfFuncs)) {
- udfFuncs = udfFuncs.stream().map(t -> {
- t.setResourceName(t.getResourceName().replaceFirst(originFullName, matcherFullName));
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- udfFunctionMapper.batchUpdateUdfFunc(udfFuncs);
- }
- }
- }
- } else if (ResourceType.UDF.equals(resource.getType())) {
- List<UdfFunc> udfFuncs = udfFunctionMapper.listUdfByResourceId(new Integer[]{resourceId});
- if (CollectionUtils.isNotEmpty(udfFuncs)) {
- udfFuncs = udfFuncs.stream().map(t -> {
- t.setResourceName(fullName);
- t.setUpdateTime(now);
- return t;
- }).collect(Collectors.toList());
- udfFunctionMapper.batchUpdateUdfFunc(udfFuncs);
- }
+ // if name unchanged, return directly without moving on HDFS
+ if (originResourceName.equals(name) && file == null) {
+ return result;
+ }
+ List<ResourcesTask> existResourcesList;
+ if (resource.isDirectory()) {
+ existResourcesList = resourceTaskMapper.selectSubfoldersFullNames(originFullName + FOLDER_SEPARATOR);
+ } else {
+ existResourcesList = resourceTaskMapper.selectByMap(
+ Collections.singletonMap("full_name", originFullName));
+ }
+
+ if (existResourcesList.size() > 0 && !fullName.equals(originFullName)) {
+ // check if any related task is online. If it is, it can not be updated.
+ for (ResourcesTask existResource : existResourcesList) {
+ int taskId = existResource.getTaskId();
+ if (processService.isTaskOnline(taskDefinitionMapper.selectById(taskId).getCode())) {
+ logger.error("can't be updated,because it is used of process definition that's online");
+ logger.error("resource task relation id:{} is used of task code {}", existResource.getId(),
+ taskDefinitionMapper.selectById(taskId).getCode());
+ putMsg(result, Status.RESOURCE_IS_USED);
+ return result;
+ }
}
- putMsg(result, Status.SUCCESS);
- Map<String, Object> resultMap = new HashMap<>();
- for (Map.Entry<Object, Object> entry : new BeanMap(resource).entrySet()) {
- if (!Constants.CLASS.equalsIgnoreCase(entry.getKey().toString())) {
- resultMap.put(entry.getKey().toString(), entry.getValue());
+ for (ResourcesTask existResource : existResourcesList) {
+ int taskId = existResource.getTaskId();
+ long taskCode = taskDefinitionMapper.selectById(taskId).getCode();
+
+ List<ProcessTaskRelation> processTaskRelation = processTaskRelationMapper.selectByMap(
+ Collections.singletonMap("post_task_code", taskCode));
+ if (processTaskRelation.size() > 0) {
+ long processDefinitionCode = processTaskRelation.get(0).getProcessDefinitionCode();
+ int processDefinitionVersion = processTaskRelation.get(0).getProcessDefinitionVersion();
+ List<ProcessTaskRelation> taskRelationList = processTaskRelationMapper.queryByProcessCode(
+ processTaskRelation.get(0).getProjectCode(),
+ processDefinitionCode);
+
+ List<TaskDefinition> taskDefinitionLogList = new ArrayList<>();
+
+ if (taskRelationList.size() > 0) {
+ ProcessDefinitionLog processDefinition =
+ processDefinitionLogMapper.queryByDefinitionCodeAndVersion(
+ processDefinitionCode, processDefinitionVersion);
+ for (ProcessTaskRelation taskRelation : taskRelationList) {
+ long taskCodeInProcess = taskRelation.getPostTaskCode();
+ TaskDefinition taskDefinition = taskDefinitionMapper.queryByCode(taskCodeInProcess);
+ if (taskCodeInProcess == taskCode) {
+ // originFullName is a prefix if isDirectory is true
+ taskDefinition.setTaskParams(RemoveResourceFromResourceList(originFullName,
+ taskDefinition.getTaskParams(),
+ resource.isDirectory()));
+ // if isDirectory is true, fullName is the new prefix. we replace old prefix
+ // of resource fullname with the new prefix.
+ // if isDirectory is false, fullName is the new path.
+ taskDefinition.setTaskParams(AddResourceToResourceList(originFullName,
+ fullName,
+ existResource.getFullName(),
+ taskDefinition.getTaskParams(),
+ resource.isDirectory()));
Review Comment:
> Why call `taskDefinition.setTaskParams` twice?
The first setTaskParams removes resources deselected by users. (it was selected as first, and then get deselected)
The second setTaskParams add resources get selected by users.
I used two setTaskParams to add code readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1004552114
##########
dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml:
##########
@@ -191,6 +191,22 @@
WHERE td.resource_ids is not null and td.resource_ids != '' and td.user_id = #{userId}
</select>
+ <select id="queryResourcesInUseWithinResourceList" resultType="java.util.HashMap">
+ SELECT distinct pd.code,td.resource_ids
+ FROM t_ds_process_task_relation ptr
+ join t_ds_process_definition pd
+ on ptr.process_definition_code=pd.code and ptr.process_definition_version = pd.version
+ and ptr.project_code=pd.project_code and pd.release_state = 1
+ join t_ds_task_definition td
+ on (ptr.pre_task_code=td.code and ptr.pre_task_version=td.version)
+ or (ptr.post_task_code=td.code and ptr.post_task_version=td.version)
+ WHERE 1 = 1
+ and
+ <foreach item="item" index="index" collection="resourcesArray" separator=" OR " >
+ td.resource_ids LIKE '%${item}%' and td.resource_ids != ''
+ </foreach>
+ </select>
Review Comment:
> This sql is complex, please split it.
> Avoid using `or` query, it will miss the index.
Right. This is part of the old logic to check what resources are currently used by tasks. This logic were actually modified in this pull request, therefore I delete it because this sql is not in use anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#issuecomment-1291526537
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12076)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY) [17 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL) [87 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12076&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list) [33.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list) [12.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12076&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on a diff in pull request #12076: [Feature-10495][Resource Center] Resource Center Refactor
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on code in PR #12076:
URL: https://github.com/apache/dolphinscheduler/pull/12076#discussion_r1005164211
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java:
##########
@@ -171,6 +183,26 @@ public String getResourceFileName(String tenantCode, String fileName) {
return String.format(FORMAT_S_S, getS3ResDir(tenantCode), fileName);
}
+ @Override
+ public String getResourceFileName(String fullName) {
Review Comment:
> It seems the same method `getResourceFileName` in `HadoopUtils`?
ya for now, it is.
In the future release, I should come up with a better way to get file name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org