You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "narrowizard (via GitHub)" <gi...@apache.org> on 2025/08/05 07:48:49 UTC
[I] [Bug] DevLake Pipeline Status Change Notification Issue: Duplicate TASK_CREATED and Missing TASK_RUNNING Notifications [incubator-devlake]
narrowizard opened a new issue, #8525:
URL: https://github.com/apache/incubator-devlake/issues/8525
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
The pipeline status change notification mechanism in DevLake is not behaving as expected. We observed that for a specific pipeline (e.g., pipeline ID 2976 as shown in the logs), the `TASK_CREATED` notification was sent twice, while the subsequent `TASK_RUNNING` notification was entirely missed. The pipeline then directly transitioned to `TASK_COMPLETED` notification. This deviates from the expected notification flow where `TASK_CREATED`, `TASK_RUNNING`, and `TASK_COMPLETED` notifications are sent sequentially as the pipeline progresses through these states.
<img width="748" height="314" alt="Image" src="https://github.com/user-attachments/assets/59a5e188-3255-42cc-818f-668aea8ed692" />
### What do you expect to happen
We expect the pipeline status change notifications to be sent accurately and uniquely for each distinct status change. Specifically, for a typical pipeline execution, we anticipate receiving notifications in the following order: `TASK_CREATED`, `TASK_RUNNING`, and `TASK_COMPLETED` (or other relevant terminal states like `TASK_FAILED`). There should be no duplicate notifications for the same status, and no intermediate status notifications should be skipped.
### How to reproduce
1. Run a DevLake pipeline.
2. Monitor the pipeline status change notifications (e.g., by observing the callback logs or notification recipients).
3. Observe that for some pipelines, `TASK_CREATED` notifications are sent twice, and `TASK_RUNNING` notifications are skipped, directly proceeding to `TASK_COMPLETED`.
### Anything else
**Investigation/Root Cause:**
Preliminary investigation suggests that this issue might be caused by the notification being sent before the pipeline status change transaction is fully committed to the database. This could lead to reading an uncommitted state or triggering the notification logic prematurely, resulting in duplicate or missed status updates in the notification stream.
Relative code
https://github.com/apache/incubator-devlake/blob/2d8cca1a43b40a4a023f538fd79143d638e7b53b/backend/server/services/pipeline.go#L297-L301
### Version
main
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] DevLake Pipeline Status Change Notification Issue: Duplicate TASK_CREATED and Missing TASK_RUNNING Notifications [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh closed issue #8525: [Bug] DevLake Pipeline Status Change Notification Issue: Duplicate TASK_CREATED and Missing TASK_RUNNING Notifications
URL: https://github.com/apache/incubator-devlake/issues/8525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] DevLake Pipeline Status Change Notification Issue: Duplicate TASK_CREATED and Missing TASK_RUNNING Notifications [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8525:
URL: https://github.com/apache/incubator-devlake/issues/8525#issuecomment-3153956168
<!-- Answer -->
You're correct about the root cause: the notification for pipeline status changes is sent asynchronously right after updating the pipeline status in the database, but before the transaction is guaranteed to be fully committed. This can lead to race conditions where notifications are sent for uncommitted or intermediate states, causing duplicates or missing notifications, as you've observed. There are currently no safeguards in the code to ensure notifications are only sent after the transaction is committed, nor is there deduplication logic to prevent multiple notifications for the same status [reference](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/server/services/pipeline.go).
No recent PRs address this, and while there are related feature requests and discussions about notification improvements, this specific bug hasn't been reported before [feature request context](https://github.com/apache/incubator-devlake/issues/8408).
To resolve this, you’ll want to ensure that notifications are only sent after the transaction is fully committed—possibly by moving notification logic outside the transaction or by using a post-commit hook if your transaction helper supports it. Adding deduplication logic to prevent repeated notifications for the same status would also help. Let me know if you want to discuss implementation details or need help navigating the relevant parts of the code.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/74b6200e-54e4-4bec-ba44-92614d48d8dc?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=githu
b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8525)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org