You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "AlanZQ (via GitHub)" <gi...@apache.org> on 2024/12/05 02:52:10 UTC
[I] [Bug] [Module Name] 包含多个子流程时，子流程实例中的任务实例获取前置任务实例参数重复问题 [dolphinscheduler]
AlanZQ opened a new issue, #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
在工作流中，创建多个子流程，例如创建12个子流程，子流程在获取上游任务参数时，会出现覆盖/错误/以及12个节点拿到的参数完全一致，这不符合预期
通过数据库中查询来看，生成的12的子流程实例，参数没有问题，但是在varpool的设置阶段，获取parent_process_instance_id所对应的varpool，是相同的，这导致子流程实例中的任务实例，执行时的参数永远是相同的
<img width="1009" alt="iShot_2024-12-05_10 44 15" src="https://github.com/user-attachments/assets/073be1eb-916b-4344-bfab-f5cb3b7dba95">
### What you expected to happen
通过数据库中查询来看，生成的12的子流程实例，参数没有问题，但是在varpool的设置阶段，获取parent_process_instance_id所对应的varpool，是相同的，这导致子流程实例中的任务实例，执行时的参数永远是相同的
通过debug setSubProcessParam这个方法，能够看出每个子流程任务实例获取到的varpool参数都有问题，并且会随机出现重复
### How to reproduce
我已经验证了3.2.1的standalone镜像，3.2.2的standalone镜像，debug镜像代码，都存在此问题，也可以在这两个镜像web服务页面，复现。我提供了我的验证工作流json，⚠️注意：导入后需要更改主流程中子流程的引用为新的id
[workflow_1733366901095.json](https://github.com/user-attachments/files/18016619/workflow_1733366901095.json)
### Anything else
_No response_
### Version
3.2.x
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2521908232
> Please using english to describe.
In the workflow, create multiple sub-processes, for example, create 12 sub-processes. When the sub-process obtains the parameters of the upstream task, there will be coverage/error/ and the parameters obtained by the 12 nodes are exactly the same, which is not as expected.
Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same
<img width="1009" alt="iShot_2024-12-05_10 44 15" src="https://github.com/user-attachments/assets/b8f4b451-065a-4212-809e-7dc4944cce61">
I have verified that this problem exists in the standalone image of 3.2.1, the standalone image of 3.2.2, and the debug image code. It can also be reproduced on the two image web service pages. I provided my verification workflow json, ⚠️Note: After importing, you need to change the reference of the sub-process in the main process to the new id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2553031867
> > > > > Is this issue fixed? I will reopen this issue.
> > > > > No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.
> > >
> > >
> > > I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all `varpool` are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.
> >
> >
> > Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.
>
> I also hope this feature can be refactored, the varpool generated should contain the workflowName and taskName. If we want to use a parameterA generate by workflowA taskA, then we should can `${workflowA.taskA.parameterA}`, then the parameter will be unique.
Got it, and thank you for your explanation. Based on my current experience, this issue does not occur when using a single sub-process. In my scenario, I modified the implementation of multiple sub-processes to use a single sub-process, which already meets the requirements of my use case. For this functionality, I hope there will be a better solution in the future.
<img width="511" alt="image" src="https://github.com/user-attachments/assets/c7971e0d-9ea9-4cce-adf9-20f1b0d62191" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2535652865
Is this issue fixed? I will reopen this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2540864210
> Is this issue fixed? I will reopen this issue.
No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2522015393
> Can you try `dev` to see if this issue exists?
ok, I will try later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2547580274
> > > Is this issue fixed? I will reopen this issue.
> > > No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.
>
> I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all `varpool` are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.
Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2518976127
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
In the workflow, create multiple sub-processes, for example, create 12 sub-processes. When the sub-process obtains the parameters of the upstream task, there will be coverage/error/ and the parameters obtained by the 12 nodes are exactly the same, which is not as expected.
Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same
<img width="1009" alt="iShot_2024-12-05_10 44 15" src="https://github.com/user-attachments/assets/073be1eb-916b-4344-bfab-f5cb3b7dba95">
### What you expected to happen
Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same
Through the debug setSubProcessParam method, it can be seen that there is a problem with the varpool parameters obtained by each sub-process task instance, and duplicates occur randomly.
### How to reproduce
I have verified that this problem exists in the standalone image of 3.2.1, the standalone image of 3.2.2, and the debug image code. It can also be reproduced on the two image web service pages. I provided my verification workflow json, ⚠️Note: After importing, you need to change the reference of the sub-process in the main process to the new id
[workflow_1733366901095.json](https://github.com/user-attachments/files/18016619/workflow_1733366901095.json)
### Anything else
_No response_
### Version
3.2.x
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2521931408
Can you try `dev` to see if this issue exists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-5358259993
I would like to work on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2552607837
> > > > Is this issue fixed? I will reopen this issue.
> > > > No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.
> >
> >
> > I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all `varpool` are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.
>
> Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.
I also hope this feature can be refactored, the varpool generated should contain the workflowName and taskName. If we want to use a parameterA generate by workflowA taskA, then we should can `${workflowA.taskA.parameterA}`, then the parameter will be unique.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #16879: [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance.
URL: https://github.com/apache/dolphinscheduler/issues/16879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2520264081
Please using english to describe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2522014772
> Can you try `dev` to see if this issue exists?
sure, off course
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "AlanZQ (via GitHub)" <gi...@apache.org>.
AlanZQ commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2527092734
> Can you try `dev` to see if this issue exists?
Encountered an error logging into the dev image.
<img width="1082" alt="image" src="https://github.com/user-attachments/assets/00dff9a5-a099-4ba6-9983-e9f22b08fd6e">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance. [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16879:
URL: https://github.com/apache/dolphinscheduler/issues/16879#issuecomment-2541536749
> > Is this issue fixed? I will reopen this issue.
> > No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.
I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all `varpool` are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org