You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/04/26 03:31:57 UTC

[GitHub] [dolphinscheduler] lenboo opened a new pull request #5390: [Feature-4417] Process definition json split

lenboo opened a new pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390


   close #4417  
   
   1. New tables and data structures, DB query interface
   2. Refactoring api
   3. master/worker refactoring
   4. Modify the front part


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] qingzhongli commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
qingzhongli commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-828397975


   > > > > Are the following changes necessary ?
   > > > > `dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/TaskType.java dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/TaskParametersUtils.java`
   > > > 
   > > > 
   > > > I think this is necessary. The modification is for task plug-in extension
   > > 
   > > 
   > > Is it difficult to extend task plug-in before the modification? can you explain it in detail?
   > 
   > i think it's better to use string to express the task-type, instead of the enum.
   > if you want a new task type, you just write the new type in the configuration, without modifying the code.
   
   Got it. Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#discussion_r630928212



##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessDefinitionController.java
##########
@@ -502,72 +491,69 @@ public Result viewTree(@ApiIgnore @RequestAttribute(value = Constants.SESSION_US
                            @ApiParam(name = "projectName", value = "PROJECT_NAME", required = true) @PathVariable String projectName,
                            @RequestParam("processId") Integer id,
                            @RequestParam("limit") Integer limit) throws Exception {
-
         Map<String, Object> result = processDefinitionService.viewTree(id, limit);
         return returnDataList(result);
     }
 
     /**
-     * get tasks list by process definition id
+     * get tasks list by process definition code
      *
-     * @param loginUser           login user
-     * @param projectName         project name
-     * @param processDefinitionId process definition id
+     * @param loginUser login user
+     * @param projectName project name
+     * @param processDefinitionCode process definition code
      * @return task list
      */
-    @ApiOperation(value = "getNodeListByDefinitionId", notes = "GET_NODE_LIST_BY_DEFINITION_ID_NOTES")
+    @ApiOperation(value = "getNodeListByDefinitionCode", notes = "GET_NODE_LIST_BY_DEFINITION_CODE_NOTES")
     @ApiImplicitParams({
-        @ApiImplicitParam(name = "processDefinitionId", value = "PROCESS_DEFINITION_ID", required = true, dataType = "Int", example = "100")
+            @ApiImplicitParam(name = "processDefinitionCode", value = "PROCESS_DEFINITION_CODE", required = true, dataType = "Long", example = "100")
     })
     @GetMapping(value = "gen-task-list")
     @ResponseStatus(HttpStatus.OK)
     @ApiException(GET_TASKS_LIST_BY_PROCESS_DEFINITION_ID_ERROR)
-    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
-    public Result getNodeListByDefinitionId(
-        @ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
-        @ApiParam(name = "projectName", value = "PROJECT_NAME", required = true) @PathVariable String projectName,
-        @RequestParam("processDefinitionId") Integer processDefinitionId) throws Exception {
-
-        Map<String, Object> result = processDefinitionService.getTaskNodeListByDefinitionId(processDefinitionId);
+    public Result getNodeListByDefinitionCode(
+            @ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+            @ApiParam(name = "projectName", value = "PROJECT_NAME", required = true) @PathVariable String projectName,
+            @RequestParam("processDefinitionCode") Long processDefinitionCode) throws Exception {
+        logger.info("query task node name list by definitionCode, login user:{}, project name:{}, code : {}",
+                loginUser.getUserName(), projectName, processDefinitionCode);

Review comment:
       It's best to be consistent with dev  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837898954


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [113 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#discussion_r630914888



##########
File path: sql/dolphinscheduler_postgre.sql
##########
@@ -287,52 +287,146 @@ CREATE TABLE t_ds_error_command (
 DROP TABLE IF EXISTS t_ds_process_definition;
 CREATE TABLE t_ds_process_definition (
   id int NOT NULL  ,
+  code bigint NOT NULL,
   name varchar(255) DEFAULT NULL ,
   version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
   release_state int DEFAULT NULL ,
-  project_id int DEFAULT NULL ,
   user_id int DEFAULT NULL ,
-  process_definition_json text ,
-  description text ,
   global_params text ,
-  flag int DEFAULT NULL ,
   locations text ,
   connects text ,
-  warning_group_id int4 DEFAULT NULL ,
-  create_time timestamp DEFAULT NULL ,
+  warning_group_id int DEFAULT NULL ,
+  flag int DEFAULT NULL ,
   timeout int DEFAULT '0' ,
-  tenant_id int NOT NULL DEFAULT '-1' ,
+  tenant_id int DEFAULT '-1' ,
+  create_time timestamp DEFAULT NULL ,
   update_time timestamp DEFAULT NULL ,
-  modify_by varchar(36) DEFAULT '' ,
-  resource_ids varchar(64),
-  PRIMARY KEY (id),
-  CONSTRAINT process_definition_unique UNIQUE (name, project_id)
+  PRIMARY KEY (id) ,
+  CONSTRAINT process_definition_unique UNIQUE (name, project_code)
 ) ;
 
-create index process_definition_index on t_ds_process_definition (project_id,id);
-
---
--- Table structure for table t_ds_process_definition_version
---
+create index process_definition_index on t_ds_process_definition (code,id);
 
-DROP TABLE IF EXISTS t_ds_process_definition_version;
-CREATE TABLE t_ds_process_definition_version (
+DROP TABLE IF EXISTS t_ds_process_definition_log;
+CREATE TABLE t_ds_process_definition_log (
   id int NOT NULL  ,
-  process_definition_id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
   version int DEFAULT NULL ,
-  process_definition_json text ,
   description text ,
+  project_code bigint DEFAULT NULL ,
+  release_state int DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
   global_params text ,
   locations text ,
   connects text ,
-  warning_group_id int4 DEFAULT NULL,
+  warning_group_id int DEFAULT NULL ,
+  flag int DEFAULT NULL ,
+  timeout int DEFAULT '0' ,
+  tenant_id int DEFAULT '-1' ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
   create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id)
+) ;
+
+DROP TABLE IF EXISTS t_ds_task_definition;
+CREATE TABLE t_ds_task_definition (
+  id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
+  version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
+  task_type varchar(50) DEFAULT NULL ,
+  task_params text ,
+  flag int DEFAULT NULL ,
+  task_priority int DEFAULT NULL ,
+  worker_group varchar(255) DEFAULT NULL ,
+  fail_retry_times int DEFAULT NULL ,
+  fail_retry_interval int DEFAULT NULL ,
+  timeout_flag int DEFAULT NULL ,
+  timeout_notify_strategy int DEFAULT NULL ,
   timeout int DEFAULT '0' ,
-  resource_ids varchar(64),
+  delay_time int DEFAULT '0' ,
+  resource_ids varchar(255) DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id) ,
+  CONSTRAINT task_definition_unique UNIQUE (name, project_code)
+) ;
+
+create index task_definition_index on t_ds_task_definition (project_code,id);
+
+DROP TABLE IF EXISTS t_ds_task_definition_log;
+CREATE TABLE t_ds_task_definition_log (
+  id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
+  version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
+  task_type varchar(50) DEFAULT NULL ,
+  task_params text ,
+  flag int DEFAULT NULL ,
+  task_priority int DEFAULT NULL ,
+  worker_group varchar(255) DEFAULT NULL ,
+  fail_retry_times int DEFAULT NULL ,
+  fail_retry_interval int DEFAULT NULL ,
+  timeout_flag int DEFAULT NULL ,
+  timeout_notify_strategy int DEFAULT NULL ,
+  timeout int DEFAULT '0' ,
+  delay_time int DEFAULT '0' ,
+  resource_ids varchar(255) DEFAULT NULL ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id)
+) ;
+
+DROP TABLE IF EXISTS t_ds_process_task_relation;
+CREATE TABLE t_ds_process_task_relation (
+  id int NOT NULL  ,
+  name varchar(255) DEFAULT NULL ,
+  process_definition_version int DEFAULT NULL ,
+  project_code bigint DEFAULT NULL ,
+  process_definition_code bigint DEFAULT NULL ,
+  pre_task_code bigint DEFAULT NULL ,
+  pre_task_version int DEFAULT '0' ,
+  post_task_code bigint DEFAULT NULL ,
+  post_task_version int DEFAULT '0' ,
+  condition_type int DEFAULT NULL ,
+  condition_params text ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
   PRIMARY KEY (id)
 ) ;
 
-create index process_definition_id_and_version on t_ds_process_definition_version (process_definition_id,version);
+DROP TABLE IF EXISTS t_ds_process_task_relation_log;
+CREATE TABLE t_ds_process_task_relation_log (
+  id int NOT NULL  ,
+  name varchar(255) DEFAULT NULL ,
+  process_definition_version int DEFAULT NULL ,
+  project_code bigint DEFAULT NULL ,
+  process_definition_code bigint DEFAULT NULL ,
+  pre_task_code bigint DEFAULT NULL ,
+  pre_task_version int DEFAULT '0' ,
+  post_task_code bigint DEFAULT NULL ,
+  post_task_version int DEFAULT '0' ,
+  condition_type int DEFAULT NULL ,
+  condition_params text ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,

Review comment:
       Why don’t the update_time and create_time be handed over to the database for management?Settings in the program can be confusing and easy to forget




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-839387745


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [113 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] wen-hemin commented on a change in pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
wen-hemin commented on a change in pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#discussion_r630180526



##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -704,6 +805,7 @@ CREATE TABLE `t_ds_task_instance` (
   `retry_times` int(4) DEFAULT '0' COMMENT 'task retry times',
   `pid` int(4) DEFAULT NULL COMMENT 'pid of task',
   `app_link` text COMMENT 'yarn app id',
+  `task_params` text COMMENT 'job custom parameters',

Review comment:
       text -> longtext

##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -391,59 +391,162 @@ CREATE TABLE `t_ds_error_command` (
 -- ----------------------------
 DROP TABLE IF EXISTS `t_ds_process_definition`;
 CREATE TABLE `t_ds_process_definition` (
-  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'key',
-  `name` varchar(255) DEFAULT NULL COMMENT 'process definition name',
+  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'self-increasing id',
+  `code` bigint(20) NOT NULL COMMENT 'encoding',
+  `name` varchar(200) DEFAULT NULL COMMENT 'process definition name',

Review comment:
       The length of the field should be the same as before.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#discussion_r630925311



##########
File path: dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/SnowFlakeUtils.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.common.utils;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Objects;
+
+public class SnowFlakeUtils {
+    // start timestamp
+    private static final long START_TIMESTAMP = 1609430400L; //2021-01-01
+    // Number of digits
+    private static final long SEQUENCE_BIT = 13;
+    private static final long MACHINE_BIT = 2;
+    private static final long MAX_SEQUENCE = ~(-1L << SEQUENCE_BIT);
+    // The displacement to the left
+    private static final long MACHINE_LEFT = SEQUENCE_BIT;
+    private static final long TIMESTAMP_LEFT = SEQUENCE_BIT + MACHINE_BIT;
+    private final int machineId;
+    private long sequence = 0L;
+    private long lastTimestamp = -1L;
+
+    private SnowFlakeUtils() throws SnowFlakeException {
+        try {
+            this.machineId = Math.abs(Objects.hash(InetAddress.getLocalHost().getHostName())) % 32;
+        } catch (UnknownHostException e) {
+            throw new SnowFlakeException(e.getMessage());

Review comment:
       Maybe it would be better to use & :) This is a boring suggestion. Not much effect. It just looks more professional.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-839387745


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [113 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] CalvinKirs merged pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837755871


   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#5390](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2dd0d93) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/d93a9dc2944856cd3d1c6cba669d30a7a44c10b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d93a9dc) will **increase** coverage by `1.22%`.
   > The diff coverage is `56.57%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/graphs/tree.svg?width=650&height=150&src=pr&token=bv9iXXRLi9&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev    #5390      +/-   ##
   ============================================
   + Coverage     45.21%   46.43%   +1.22%     
   - Complexity     3395     3774     +379     
   ============================================
     Files           538      608      +70     
     Lines         23053    24934    +1881     
     Branches       2687     2835     +148     
   ============================================
   + Hits          10423    11579    +1156     
   - Misses        11698    12253     +555     
   - Partials        932     1102     +170     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...inscheduler/plugin/alert/email/EmailConstants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1hbGVydC1lbWFpbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vYWxlcnQvZW1haWwvRW1haWxDb25zdGFudHMuamF2YQ==) | `87.50% <ø> (ø)` | `1.00 <0.00> (ø)` | |
   | [...eduler/plugin/alert/script/ScriptAlertChannel.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1hbGVydC1zY3JpcHQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL2FsZXJ0L3NjcmlwdC9TY3JpcHRBbGVydENoYW5uZWwuamF2YQ==) | `16.66% <0.00%> (ø)` | `1.00 <0.00> (ø)` | |
   | [...org/apache/dolphinscheduler/alert/AlertServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9hbGVydC9BbGVydFNlcnZlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...che/dolphinscheduler/api/ApiApplicationServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL0FwaUFwcGxpY2F0aW9uU2VydmVyLmphdmE=) | `33.33% <0.00%> (ø)` | `1.00 <0.00> (ø)` | |
   | [...cheduler/api/controller/AccessTokenController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWNjZXNzVG9rZW5Db250cm9sbGVyLmphdmE=) | `6.66% <ø> (-1.67%)` | `1.00 <0.00> (-1.00)` | |
   | [...scheduler/api/controller/AlertGroupController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWxlcnRHcm91cENvbnRyb2xsZXIuamF2YQ==) | `8.00% <ø> (+2.73%)` | `2.00 <0.00> (ø)` | |
   | [.../api/controller/AlertPluginInstanceController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWxlcnRQbHVnaW5JbnN0YW5jZUNvbnRyb2xsZXIuamF2YQ==) | `7.69% <ø> (+1.97%)` | `2.00 <0.00> (ø)` | |
   | [...heduler/api/controller/DataAnalysisController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YUFuYWx5c2lzQ29udHJvbGxlci5qYXZh) | `9.09% <ø> (ø)` | `1.00 <0.00> (-1.00)` | |
   | [...scheduler/api/controller/DataSourceController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YVNvdXJjZUNvbnRyb2xsZXIuamF2YQ==) | `4.16% <0.00%> (-0.49%)` | `1.00 <0.00> (-1.00)` | |
   | [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (-2.78%)` | `1.00 <0.00> (-1.00)` | |
   | ... and [372 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [25727d2...2dd0d93](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-839670581


   I will merge this branch first. (Due to the special merger method)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837761194


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [114 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837755871


   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#5390](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aa8b51) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/d93a9dc2944856cd3d1c6cba669d30a7a44c10b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d93a9dc) will **increase** coverage by `1.22%`.
   > The diff coverage is `56.57%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/graphs/tree.svg?width=650&height=150&src=pr&token=bv9iXXRLi9&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev    #5390      +/-   ##
   ============================================
   + Coverage     45.21%   46.43%   +1.22%     
   - Complexity     3395     3773     +378     
   ============================================
     Files           538      608      +70     
     Lines         23053    24934    +1881     
     Branches       2687     2835     +148     
   ============================================
   + Hits          10423    11578    +1155     
   - Misses        11698    12253     +555     
   - Partials        932     1103     +171     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...inscheduler/plugin/alert/email/EmailConstants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1hbGVydC1lbWFpbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vYWxlcnQvZW1haWwvRW1haWxDb25zdGFudHMuamF2YQ==) | `87.50% <ø> (ø)` | `1.00 <0.00> (ø)` | |
   | [...eduler/plugin/alert/script/ScriptAlertChannel.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1hbGVydC1zY3JpcHQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL2FsZXJ0L3NjcmlwdC9TY3JpcHRBbGVydENoYW5uZWwuamF2YQ==) | `16.66% <0.00%> (ø)` | `1.00 <0.00> (ø)` | |
   | [...org/apache/dolphinscheduler/alert/AlertServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9hbGVydC9BbGVydFNlcnZlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...che/dolphinscheduler/api/ApiApplicationServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL0FwaUFwcGxpY2F0aW9uU2VydmVyLmphdmE=) | `33.33% <0.00%> (ø)` | `1.00 <0.00> (ø)` | |
   | [...cheduler/api/controller/AccessTokenController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWNjZXNzVG9rZW5Db250cm9sbGVyLmphdmE=) | `6.66% <ø> (-1.67%)` | `1.00 <0.00> (-1.00)` | |
   | [...scheduler/api/controller/AlertGroupController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWxlcnRHcm91cENvbnRyb2xsZXIuamF2YQ==) | `8.00% <ø> (+2.73%)` | `2.00 <0.00> (ø)` | |
   | [.../api/controller/AlertPluginInstanceController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQWxlcnRQbHVnaW5JbnN0YW5jZUNvbnRyb2xsZXIuamF2YQ==) | `7.69% <ø> (+1.97%)` | `2.00 <0.00> (ø)` | |
   | [...heduler/api/controller/DataAnalysisController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YUFuYWx5c2lzQ29udHJvbGxlci5qYXZh) | `9.09% <ø> (ø)` | `1.00 <0.00> (-1.00)` | |
   | [...scheduler/api/controller/DataSourceController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YVNvdXJjZUNvbnRyb2xsZXIuamF2YQ==) | `4.16% <0.00%> (-0.49%)` | `1.00 <0.00> (-1.00)` | |
   | [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (-2.78%)` | `1.00 <0.00> (-1.00)` | |
   | ... and [372 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5390/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [25727d2...7aa8b51](https://codecov.io/gh/apache/dolphinscheduler/pull/5390?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837898954


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [113 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] JinyLeeChina commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-826830480


   > Are the following changes necessary ?
   > `dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/TaskType.java dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/TaskParametersUtils.java`
   
   I think this is necessary. The modification is for task plug-in extension


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] lenboo commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
lenboo commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-828098183


   > > > Are the following changes necessary ?
   > > > `dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/TaskType.java dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/TaskParametersUtils.java`
   > > 
   > > 
   > > I think this is necessary. The modification is for task plug-in extension
   > 
   > Is it difficult to extend task plug-in before the modification? can you explain it in detail?
   
   i think it's better to use string to express the task-type,  instead of the enum.  
   if you want a new task type, you just write the new type in the configuration, without modifying the code.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-839608888


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [113 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] lenboo commented on a change in pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
lenboo commented on a change in pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#discussion_r630921994



##########
File path: sql/dolphinscheduler_postgre.sql
##########
@@ -287,52 +287,146 @@ CREATE TABLE t_ds_error_command (
 DROP TABLE IF EXISTS t_ds_process_definition;
 CREATE TABLE t_ds_process_definition (
   id int NOT NULL  ,
+  code bigint NOT NULL,
   name varchar(255) DEFAULT NULL ,
   version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
   release_state int DEFAULT NULL ,
-  project_id int DEFAULT NULL ,
   user_id int DEFAULT NULL ,
-  process_definition_json text ,
-  description text ,
   global_params text ,
-  flag int DEFAULT NULL ,
   locations text ,
   connects text ,
-  warning_group_id int4 DEFAULT NULL ,
-  create_time timestamp DEFAULT NULL ,
+  warning_group_id int DEFAULT NULL ,
+  flag int DEFAULT NULL ,
   timeout int DEFAULT '0' ,
-  tenant_id int NOT NULL DEFAULT '-1' ,
+  tenant_id int DEFAULT '-1' ,
+  create_time timestamp DEFAULT NULL ,
   update_time timestamp DEFAULT NULL ,
-  modify_by varchar(36) DEFAULT '' ,
-  resource_ids varchar(64),
-  PRIMARY KEY (id),
-  CONSTRAINT process_definition_unique UNIQUE (name, project_id)
+  PRIMARY KEY (id) ,
+  CONSTRAINT process_definition_unique UNIQUE (name, project_code)
 ) ;
 
-create index process_definition_index on t_ds_process_definition (project_id,id);
-
---
--- Table structure for table t_ds_process_definition_version
---
+create index process_definition_index on t_ds_process_definition (code,id);
 
-DROP TABLE IF EXISTS t_ds_process_definition_version;
-CREATE TABLE t_ds_process_definition_version (
+DROP TABLE IF EXISTS t_ds_process_definition_log;
+CREATE TABLE t_ds_process_definition_log (
   id int NOT NULL  ,
-  process_definition_id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
   version int DEFAULT NULL ,
-  process_definition_json text ,
   description text ,
+  project_code bigint DEFAULT NULL ,
+  release_state int DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
   global_params text ,
   locations text ,
   connects text ,
-  warning_group_id int4 DEFAULT NULL,
+  warning_group_id int DEFAULT NULL ,
+  flag int DEFAULT NULL ,
+  timeout int DEFAULT '0' ,
+  tenant_id int DEFAULT '-1' ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
   create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id)
+) ;
+
+DROP TABLE IF EXISTS t_ds_task_definition;
+CREATE TABLE t_ds_task_definition (
+  id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
+  version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
+  task_type varchar(50) DEFAULT NULL ,
+  task_params text ,
+  flag int DEFAULT NULL ,
+  task_priority int DEFAULT NULL ,
+  worker_group varchar(255) DEFAULT NULL ,
+  fail_retry_times int DEFAULT NULL ,
+  fail_retry_interval int DEFAULT NULL ,
+  timeout_flag int DEFAULT NULL ,
+  timeout_notify_strategy int DEFAULT NULL ,
   timeout int DEFAULT '0' ,
-  resource_ids varchar(64),
+  delay_time int DEFAULT '0' ,
+  resource_ids varchar(255) DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id) ,
+  CONSTRAINT task_definition_unique UNIQUE (name, project_code)
+) ;
+
+create index task_definition_index on t_ds_task_definition (project_code,id);
+
+DROP TABLE IF EXISTS t_ds_task_definition_log;
+CREATE TABLE t_ds_task_definition_log (
+  id int NOT NULL  ,
+  code bigint NOT NULL,
+  name varchar(255) DEFAULT NULL ,
+  version int DEFAULT NULL ,
+  description text ,
+  project_code bigint DEFAULT NULL ,
+  user_id int DEFAULT NULL ,
+  task_type varchar(50) DEFAULT NULL ,
+  task_params text ,
+  flag int DEFAULT NULL ,
+  task_priority int DEFAULT NULL ,
+  worker_group varchar(255) DEFAULT NULL ,
+  fail_retry_times int DEFAULT NULL ,
+  fail_retry_interval int DEFAULT NULL ,
+  timeout_flag int DEFAULT NULL ,
+  timeout_notify_strategy int DEFAULT NULL ,
+  timeout int DEFAULT '0' ,
+  delay_time int DEFAULT '0' ,
+  resource_ids varchar(255) DEFAULT NULL ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
+  PRIMARY KEY (id)
+) ;
+
+DROP TABLE IF EXISTS t_ds_process_task_relation;
+CREATE TABLE t_ds_process_task_relation (
+  id int NOT NULL  ,
+  name varchar(255) DEFAULT NULL ,
+  process_definition_version int DEFAULT NULL ,
+  project_code bigint DEFAULT NULL ,
+  process_definition_code bigint DEFAULT NULL ,
+  pre_task_code bigint DEFAULT NULL ,
+  pre_task_version int DEFAULT '0' ,
+  post_task_code bigint DEFAULT NULL ,
+  post_task_version int DEFAULT '0' ,
+  condition_type int DEFAULT NULL ,
+  condition_params text ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,
   PRIMARY KEY (id)
 ) ;
 
-create index process_definition_id_and_version on t_ds_process_definition_version (process_definition_id,version);
+DROP TABLE IF EXISTS t_ds_process_task_relation_log;
+CREATE TABLE t_ds_process_task_relation_log (
+  id int NOT NULL  ,
+  name varchar(255) DEFAULT NULL ,
+  process_definition_version int DEFAULT NULL ,
+  project_code bigint DEFAULT NULL ,
+  process_definition_code bigint DEFAULT NULL ,
+  pre_task_code bigint DEFAULT NULL ,
+  pre_task_version int DEFAULT '0' ,
+  post_task_code bigint DEFAULT NULL ,
+  post_task_version int DEFAULT '0' ,
+  condition_type int DEFAULT NULL ,
+  condition_params text ,
+  operator int DEFAULT NULL ,
+  operate_time timestamp DEFAULT NULL ,
+  create_time timestamp DEFAULT NULL ,
+  update_time timestamp DEFAULT NULL ,

Review comment:
       not all database can manage the time, and we will support other databases in the future.
   so we can manage it ourself.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-837761194


   SonarCloud Quality Gate failed.
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=BUG)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY) [7 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=VULNERABILITY)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=SECURITY_HOTSPOT)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL) [114 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5390&resolved=false&types=CODE_SMELL)
   
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/25.png' alt='38.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_coverage&view=list)  
   [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.3%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list) [2.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5390&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] qingzhongli commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
qingzhongli commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-827909848


   > > Are the following changes necessary ?
   > > `dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/TaskType.java dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/TaskParametersUtils.java`
   > 
   > I think this is necessary. The modification is for task plug-in extension
   
   Is it difficult to extend task plug-in before the modification? can you explain it in detail?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] lenboo commented on pull request #5390: [Feature-4417] Process definition json split

Posted by GitBox <gi...@apache.org>.
lenboo commented on pull request #5390:
URL: https://github.com/apache/dolphinscheduler/pull/5390#issuecomment-839377684


   > Please look comment
   
   thank you very much for the review, i will fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org