You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "chempin (via GitHub)" <gi...@apache.org> on 2023/02/27 06:37:30 UTC
[GitHub] [dolphinscheduler] chempin opened a new issue, #13637: [Bug] [processInstance globalParams lost] 3.1.3version
chempin opened a new issue, #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
- when process instance running with command CommandType.RECOVER_SERIAL_WAIT, and process instance GlobalParams will lost it value like this [{"prop":"paramName","direct":"IN","type":"VARCHAR","value":""}]
### What you expected to happen
do not lost it value.
### How to reproduce
1、 create a new porcess with "serial wait" policy.
2、start 2 process instance, and make the second process instance in "serial wait" state,check the second process instance GlobalParams
3、wait for he second process instance running with command CommandType.RECOVER_SERIAL_WAIT, and check the second process instance GlobalParams again ,it changed and lost it value
### Anything else
_No response_
### Version
3.1.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1471439644
<img width="865" alt="图片" src="https://user-images.githubusercontent.com/5117373/225543632-aba580c4-f6e3-4978-bcec-ee04ec7e8d1e.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] 1205747671 commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "1205747671 (via GitHub)" <gi...@apache.org>.
1205747671 commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1639591229
> ```java
> private static void hanlerSerialGlobalPrams(ProcessInstance processInstance) {
> // 不知道串行等待的时候 global_params 对不对
> // 判断下 command_params 是否为空
> // get the startParams user specified at the first starting while repeat running is needed
> Map<String, Object> commandMap =
> JSONUtils.parseObject(processInstance.getCommandParam(), new TypeReference<Map<String, Object>>() {
> });
> Map<String, String> startParamMap = null;
> if (MapUtils.isNotEmpty(commandMap)) {
> Object startParamsJson = commandMap.get(CMD_PARAM_START_PARAMS);
> if (startParamsJson != null) {
> String startParams = startParamsJson.toString();
> startParamMap = JSONUtils.toMap(startParams);
> }
> }
> if (startParamMap != null && startParamMap.size() > 0) {
> // commandParams
> List<Map<String, String>> globalParamsList = new ArrayList<>();
> startParamMap.forEach((key, value) -> {
> Map<String, String> globalParamsMap = new HashMap<>();
> globalParamsMap.put("prop", key);
> globalParamsMap.put("direct", "IN");
> globalParamsMap.put("type", "VARCHAR");
> globalParamsMap.put("value", value);
> globalParamsList.add(globalParamsMap);
> });
>
> processInstance.setGlobalParams(JSONUtils.toJsonString(globalParamsList));
> }
> }
> ```
If you write it this way, the startParam cannot contain built-in parameters. Otherwise, they will be replaced by expressions of built-in parameters in the subsequent process, such as "$[yyyyMMddHHmmss]". Hence, the 'date' in the following process cannot be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1477211553
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "ljjjjjjjjjjjjjjjjjjjjj (via GitHub)" <gi...@apache.org>.
ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1465476783
@Override
@Transactional
public ProcessInstance handleCommand(String host,
Command command) throws CronParseException, CodeGenerateException {
ProcessInstance processInstance = **constructProcessInstance**(command, host);
// cannot construct process instance, return null
if (processInstance == null) {
logger.error("scan command, command parameter is error: {}", command);
moveToErrorCommand(command, "process instance is null");
return null;
}
processInstance.setCommandType(command.getCommandType());
processInstance.addHistoryCmd(command.getCommandType());
// if the processDefinition is serial
ProcessDefinition processDefinition = this.findProcessDefinition(processInstance.getProcessDefinitionCode(),
processInstance.getProcessDefinitionVersion());
if (processDefinition.getExecutionType().typeIsSerial()) {
saveSerialProcess(processInstance, processDefinition);
if (processInstance.getState() != WorkflowExecutionStatus.SUBMITTED_SUCCESS) {
setSubProcessParam(processInstance);
deleteCommandWithCheck(command.getId());
return null;
}
} else {
processInstanceDao.upsertProcessInstance(processInstance);
}
setSubProcessParam(processInstance);
deleteCommandWithCheck(command.getId());
return processInstance;
}
protected @Nullable ProcessInstance constructProcessInstance(Command command,
String host) throws CronParseException, CodeGenerateException {
ProcessInstance processInstance;
ProcessDefinition processDefinition;
CommandType commandType = command.getCommandType();
processDefinition =
this.findProcessDefinition(command.getProcessDefinitionCode(), command.getProcessDefinitionVersion());
if (processDefinition == null) {
logger.error("cannot find the work process define! define code : {}", command.getProcessDefinitionCode());
throw new IllegalArgumentException("Cannot find the process definition for this workflowInstance");
}
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
int processInstanceId = command.getProcessInstanceId();
if (processInstanceId == 0) {
processInstance = generateNewProcessInstance(processDefinition, command, cmdParam);
} else {
processInstance = this.findProcessInstanceDetailById(processInstanceId).orElse(null);
if (processInstance == null) {
return null;
}
}
if (cmdParam != null) { **//this is the point, RECOVER_SERIAL_WAIT command do not have the orignal params, and deleted global params in the record**
CommandType commandTypeIfComplement = getCommandTypeIfComplement(processInstance, command);
// reset global params while repeat running is needed by cmdParam
if (commandTypeIfComplement == CommandType.REPEAT_RUNNING) {
setGlobalParamIfCommanded(processDefinition, cmdParam);
}
// time zone
String timezoneId = cmdParam.get(Constants.SCHEDULE_TIMEZONE);
// Recalculate global parameters after rerun.
String globalParams = curingGlobalParamsService.curingGlobalParams(processInstance.getId(),
processDefinition.getGlobalParamMap(),
processDefinition.getGlobalParamList(),
commandTypeIfComplement,
processInstance.getScheduleTime(), timezoneId);
processInstance.setGlobalParams(globalParams);
processInstance.setProcessDefinition(processDefinition);
}
// reset command parameter
if (processInstance.getCommandParam() != null) {
Map<String, String> processCmdParam = JSONUtils.toMap(processInstance.getCommandParam());
processCmdParam.forEach((key, value) -> {
if (!cmdParam.containsKey(key)) {
cmdParam.put(key, value);
}
});
}
// reset command parameter if sub process
if (cmdParam.containsKey(CommandKeyConstants.CMD_PARAM_SUB_PROCESS)) {
processInstance.setCommandParam(command.getCommandParam());
}
if (Boolean.FALSE.equals(checkCmdParam(command, cmdParam))) {
logger.error("command parameter check failed!");
return null;
}
if (command.getScheduleTime() != null) {
processInstance.setScheduleTime(command.getScheduleTime());
}
processInstance.setHost(host);
processInstance.setRestartTime(new Date());
WorkflowExecutionStatus runStatus = WorkflowExecutionStatus.RUNNING_EXECUTION;
int runTime = processInstance.getRunTimes();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "ljjjjjjjjjjjjjjjjjjjjj (via GitHub)" <gi...@apache.org>.
ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1471488890
so, with method(hanlerSerialGlobalPrams), processInstance GlobalParams rebuild from command params ,is it right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1471437712
this code is my wirte , we test ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1445787595
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "ljjjjjjjjjjjjjjjjjjjjj (via GitHub)" <gi...@apache.org>.
ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1465505930
which version could i find this method(hanlerSerialGlobalPrams)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1447426444
waitting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "ljjjjjjjjjjjjjjjjjjjjj (via GitHub)" <gi...@apache.org>.
ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1465503774
This code is part of method constructProcessInstance in ProcessServiceImpl.java
with the log I replay before ,ProcessInstance 53 alreday exists in t_ds_process_instance. It created by command 100, and there are some value in global params.
When command 101 execute , program will recreate process_instance 53 , due to the following code, global params lost
RECOVER_SERIAL_WAIT command -> handleCommand -> constructProcessInstance -> lost global params.
if (cmdParam != null) {
CommandType commandTypeIfComplement = getCommandTypeIfComplement(processInstance, command);
// reset global params while repeat running is needed by cmdParam
if (commandTypeIfComplement == CommandType.REPEAT_RUNNING) {
setGlobalParamIfCommanded(processDefinition, cmdParam);
}
// time zone
String timezoneId = cmdParam.get(Constants.SCHEDULE_TIMEZONE);
// Recalculate global parameters after rerun.
String globalParams = curingGlobalParamsService.curingGlobalParams(processInstance.getId(),
processDefinition.getGlobalParamMap(),
processDefinition.getGlobalParamList(),
commandTypeIfComplement,
processInstance.getScheduleTime(), timezoneId);
processInstance.setGlobalParams(globalParams);
processInstance.setProcessDefinition(processDefinition);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "ljjjjjjjjjjjjjjjjjjjjj (via GitHub)" <gi...@apache.org>.
ljjjjjjjjjjjjjjjjjjjjj commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1465473594
API.LOG
[DEBUG] 2023-03-13 11:22:05.847 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - ==> Preparing: INSERT INTO t_ds_command (id, command_type, process_definition_code, executor_id, command_param, task_depend_type, failure_strategy, warning_type, warning_group_id, schedule_time,start_time, process_instance_priority, update_time, worker_group, environment_code, dry_run, process_instance_id, process_definition_version ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
[DEBUG] 2023-03-13 11:22:05.848 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - ==> Parameters: 99(Integer), 0(Integer), 8814727200320(Long), 1(Integer), {"StartParams":"{\"bizTime\":\"2023-03-13 11:22:05\"}","schedule_timezone":"Asia/Shanghai"}(String), 2(Integer), 1(Integer), 0(Integer), 0(Integer), null, 2023-03-13 11:22:05.838(Timestamp), 2(Integer), 2023-03-13 11:22:05.838(Timestamp), default(String), -1(Long), 0(Integer), 0(Integer), 1(Integer)
[DEBUG] 2023-03-13 11:22:05.849 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - <== Updates: 1
[DEBUG] 2023-03-13 11:22:09.604 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - ==> Preparing: INSERT INTO t_ds_command (id, command_type, process_definition_code, executor_id, command_param, task_depend_type, failure_strategy, warning_type, warning_group_id, schedule_time,start_time, process_instance_priority, update_time, worker_group, environment_code, dry_run, process_instance_id, process_definition_version ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
[DEBUG] 2023-03-13 11:22:09.604 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - ==> Parameters: 100(Integer), 0(Integer), 8814727200320(Long), 1(Integer), {"StartParams":"{\"bizTime\":\"2023-03-13 11:22:09\"}","schedule_timezone":"Asia/Shanghai"}(String), 2(Integer), 1(Integer), 0(Integer), 0(Integer), null, 2023-03-13 11:22:09.596(Timestamp), 2(Integer), 2023-03-13 11:22:09.596(Timestamp), default(String), -1(Long), 0(Integer), 0(Integer), 1(Integer)
[DEBUG] 2023-03-13 11:22:09.605 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert:[137] - <== Updates: 1
MASTER.LOG
[DEBUG] 2023-03-13 11:24:07.916 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert - ==> Preparing: INSERT INTO t_ds_command (id, command_type, process_definition_code, executor_id, command_param, task_depend_type, failure_strategy, warning_type, warning_group_id, schedule_time,start_time, process_instance_priority, update_time, worker_group, environment_code, dry_run, process_instance_id, process_definition_version ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
[DEBUG] 2023-03-13 11:24:07.916 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert - ==> Parameters: 101(Integer), 11(Integer), 8814727200320(Long), 0(Integer), {"ProcessInstanceId":"53","schedule_timezone":"Asia/Shanghai"}(String), 2(Integer), 1(Integer), null, null, null, 2023-03-13 11:24:07.914(Timestamp), null, 2023-03-13 11:24:07.914(Timestamp), null, null, 0(Integer), 53(Integer), 1(Integer)
[DEBUG] 2023-03-13 11:24:07.916 org.apache.dolphinscheduler.dao.mapper.CommandMapper.insert - <== Updates: 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1465447156
```java
private static void hanlerSerialGlobalPrams(ProcessInstance processInstance) {
// 不知道串行等待的时候 global_params 对不对
// 判断下 command_params 是否为空
// get the startParams user specified at the first starting while repeat running is needed
Map<String, Object> commandMap =
JSONUtils.parseObject(processInstance.getCommandParam(), new TypeReference<Map<String, Object>>() {
});
Map<String, String> startParamMap = null;
if (MapUtils.isNotEmpty(commandMap)) {
Object startParamsJson = commandMap.get(CMD_PARAM_START_PARAMS);
if (startParamsJson != null) {
String startParams = startParamsJson.toString();
startParamMap = JSONUtils.toMap(startParams);
}
}
if (startParamMap != null && startParamMap.size() > 0) {
// commandParams
List<Map<String, String>> globalParamsList = new ArrayList<>();
startParamMap.forEach((key, value) -> {
Map<String, String> globalParamsMap = new HashMap<>();
globalParamsMap.put("prop", key);
globalParamsMap.put("direct", "IN");
globalParamsMap.put("type", "VARCHAR");
globalParamsMap.put("value", value);
globalParamsList.add(globalParamsMap);
});
processInstance.setGlobalParams(JSONUtils.toJsonString(globalParamsList));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] niaoshuai commented on issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
Posted by "niaoshuai (via GitHub)" <gi...@apache.org>.
niaoshuai commented on issue #13637:
URL: https://github.com/apache/dolphinscheduler/issues/13637#issuecomment-1464882911
find method saveSerialProcess
And DB command_params 和 global_params is different
<img width="808" alt="图片" src="https://user-images.githubusercontent.com/5117373/224479742-08ce1223-1ca3-4aba-9fa3-28d3b1eb2d6a.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [processInstance globalParams lost] 3.1.3version [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #13637: [Bug] [processInstance globalParams lost] 3.1.3version
URL: https://github.com/apache/dolphinscheduler/issues/13637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org