You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "usharerose (via GitHub)" <gi...@apache.org> on 2025/01/02 07:28:29 UTC

[I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

usharerose opened a new issue, #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   Some merge requests themselves have been collected, but their notes (comments) are missing.
   
   It seems that only opened Merge Request would be the input of mr_notes_collector ([reference](https://github.com/apache/incubator-devlake/blob/release-v1.0/backend/plugins/gitlab/tasks/shared.go#L193)), which means the following MRs' note data would not be collected
   
   * the history MRs (closed, merged) of one GitLab project
   * one MR's complete lifecycle (created, commented, and merged) fell within the gap between DevLake pipelines
   
   ### What do you expect to happen
   
   The notes have been collected
   
   ### How to reproduce
   
   Collect any GitLab project with merged or closed MRs, those MRs' related notes would be missing.
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   v1.0
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "karpanin (via GitHub)" <gi...@apache.org>.
karpanin commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2648566880

   Yes, I have the same problem with task Collect MR Commits. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "hippojay (via GitHub)" <gi...@apache.org>.
hippojay commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2649010940

   I've been running with this commit reverted for a week - it has fixed all the Dora metrics, mr pickup times, and other stats. 
   
   It does run longer due to collecting more information - but is necessary for accurate data. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2567373234

   <!-- Answer -->
   The issue you're encountering is due to the current implementation of the `GetMergeRequestsIterator` function, which only considers open merge requests for note collection. This is intended to speed up the process by focusing on active merge requests <sup>[[1]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/gitlab/tasks/shared.go)</sup>.
   
   To address this and include notes from closed or merged merge requests, you would need to modify the `GetMergeRequestsIterator` function. Specifically, you can adjust the SQL `WHERE` condition to include "closed" and "merged" states in addition to "opened". Here's a snippet of how you can modify the code:
   
   ```go
   func GetMergeRequestsIterator(taskCtx plugin.SubTaskContext, apiCollector *api.StatefulApiCollector) (*api.DalCursorIterator, errors.Error) {
   	db := taskCtx.GetDal()
   	data := taskCtx.GetData().(*GitlabTaskData)
   	clauses := []dal.Clause{
   		dal.Select("gmr.gitlab_id, gmr.iid"),
   		dal.From("_tool_gitlab_merge_requests gmr"),
   		// collect notes and commits for opened, closed, and merged merge requests
   		dal.Where(
   			`gmr.project_id = ? and gmr.connection_id = ? AND state IN (?, ?, ?)`,
   			data.Options.ProjectId, data.Options.ConnectionId, "opened", "closed", "merged",
   		),
   	}
   	if apiCollector != nil {
   		if apiCollector.GetSince() != nil {
   			clauses = append(clauses, dal.Where("gitlab_updated_at > ?", *apiCollector.GetSince()))
   		}
   	}
   	// construct the input iterator
   	cursor, err := db.Cursor(clauses...)
   	if err != nil {
   		return nil, err
   	}
   
   	return api.NewDalCursorIterator(db, cursor, reflect.TypeOf(GitlabInput{}))
   }
   ```
   
   This change will allow the collection of notes from all states of merge requests, ensuring that historical and lifecycle-complete notes are included in the data collection process <sup>[[1]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/gitlab/tasks/shared.go)</sup>.
   
   
   <!-- Dosu Comment Footer -->
   *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
   
   ---
   <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/6b1f0fb3-b7b5-4730-90e6-509e3034065b?feedback_type=other)</sup>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2822767816

   This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "hippojay (via GitHub)" <gi...@apache.org>.
hippojay commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2622537145

   This bug breaks DORA cycle time metrics.  Only using open merge requests to collect merge commits and notes/comments means that it is not possible to calculate meaningful and actionable metrics. 
   
   It was introduced in this pull request https://github.com/apache/incubator-devlake/pull/7997 and looks to be in release 1.0.2 beta 4 onwards.
   
   Relatively easy to reverse this change through removal of the "state =" clause, which I have briefly confirmed works in a limited test env.
      


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2649716461

   The problem should be fixed by #8255 already, I think we should release a new beta version for everybody.
   @d4x1 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2803408944

   This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8255: [Bug][GitLab] Missing the notes of Merge Request
URL: https://github.com/apache/incubator-devlake/issues/8255


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][GitLab] Missing the notes of Merge Request [incubator-devlake]

Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on issue #8255:
URL: https://github.com/apache/incubator-devlake/issues/8255#issuecomment-2656166719

   We have released a new beta version [v1.0.2-beta6](https://github.com/apache/incubator-devlake/releases/tag/v1.0.2-beta6), just have a try.   @klesh @hippojay @usharerose @karpanin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org