You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "caioq (via GitHub)" <gi...@apache.org> on 2024/08/06 13:51:23 UTC
[I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
caioq opened a new issue, #7851:
URL: https://github.com/apache/incubator-devlake/issues/7851
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
I'm having some issues with getting Pull Requests using Github datasource. There are pull requests that aren't being updated completely. I noticed this when PRs that were already merged still didn't have the merged_date and closed_date fields filled in and the status was still OPEN.
I already have tried using the blueprint in advanced and default mode.
I'm using GraphQL Github api.
### What do you expect to happen
All Pull Requests being successfully updated
### How to reproduce
Create a blueprint on normal or advanced mode and check for next executions if all the pull requests are being updated.
### Anything else
Part of the logs of the stage
```
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [api async client] creating scheduler for api \"https://api.github.com/\", number of workers: 20, 14500 reqs / 1h0m0s (interval: 248.275862ms)"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] github graphql init success with remaining 12500/12500 and will reset at 2024-08-01 13:32:19 +0000 UTC"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] total step: 17"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] executing subtask Collect Pull Requests"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] start graphql collection"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] get data from _raw_github_graphql_prs where params={\"ConnectionId\":1,\"Name\":\"XXX\"} and got 39"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] existing data parser return ErrFinishCollect, but skip. rawId: #1"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] finished records: 1"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] existing data parser return ErrFinishCollect, but skip. rawId: #2"
time="2024-08-01 12:32:19" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] existing data parser return ErrFinishCollect, but skip. rawId: #3"
time="2024-08-01 12:32:20" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] collector finish by parser, rawId: #25942"
time="2024-08-01 12:32:20" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Pull Requests] ended api collection without error"
time="2024-08-01 12:32:20" level=info msg=" [pipeline service] [pipeline #36] [task #3456] finished step: 1 / 17"
time="2024-08-01 12:32:20" level=info msg=" [pipeline service] [pipeline #36] [task #3456] executing subtask Extract Pull Requests"
time="2024-08-01 12:32:20" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Extract Pull Requests] get data from _raw_github_graphql_prs where params={\"ConnectionId\":1,\"Name\":\"XXX\"} and got 40"
time="2024-08-01 12:32:21" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Extract Pull Requests] finished records: 1"
time="2024-08-01 12:32:21" level=info msg=" [pipeline service] [pipeline #36] [task #3456] finished step: 2 / 17"
time="2024-08-01 12:32:21" level=info msg=" [pipeline service] [pipeline #36] [task #3456] executing subtask Collect Users"
time="2024-08-01 12:32:21" level=info msg=" [pipeline service] [pipeline #36] [task #3456] [Collect Users] start graphql collection"
```
### Version
v1.0.1-beta5
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #7851:
URL: https://github.com/apache/incubator-devlake/issues/7851#issuecomment-2271361543
<!-- Greeting -->
Hello, @caioq! I'm here to help you with any bugs, questions, or contributions you might have. Let's work together to solve any issues you're facing.
<!-- Answer -->
The issue you're experiencing with pull requests not being updated correctly, specifically with `merged_date` and `closed_date` fields remaining empty and the status still showing as OPEN, can be due to several factors:
1. **Null Fields for Unmerged/Unclosed PRs**: The `merged_date` and `closed_date` fields are null when the PR is not recognized as merged or closed in the datasource. This can happen if the data transformation or mapping process does not correctly update these fields <sup>[[1]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v0.16/DataModels/DevLakeDomainLayerSchema.md)</sup>.
2. **GraphQL Query Fields**: Ensure that the GraphQL queries used to fetch pull request data include the necessary fields (`merged_date` and `closed_date`). The `convertGithubPullRequest` function in the code confirms that these fields are included <sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github_graphql/tasks/pr_extractor.go)</sup>.
3. **Configuration Settings**: Verify that your blueprint configurations are correct. Specific settings, such as the use of GraphQL APIs, personal access token permissions, and scope configurations, can affect data collection and extraction <sup>[[3]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v0.19/Configuration/GitHub.md)</sup>.
4. **Logs and Error Messages**: Check for any logs or error messages that might indicate issues with the data transformation or mapping process. Errors during regular expression compilation, JSON unmarshalling, or pull request conversion can lead to incomplete data updates <sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github_graphql/tasks/pr_extractor.go)</sup>.
Review these aspects to ensure that the pull request data is being correctly collected and updated. If the issue persists, further investigation into the specific configurations and logs may be necessary.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
Posted by "mdapke-insight (via GitHub)" <gi...@apache.org>.
mdapke-insight commented on issue #7851:
URL: https://github.com/apache/incubator-devlake/issues/7851#issuecomment-2469863627
@klesh are these bug fixes also patched to the lower versions? @caioq ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh closed issue #7851: [Bug][Github] Pull Requests not being updated
URL: https://github.com/apache/incubator-devlake/issues/7851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #7851:
URL: https://github.com/apache/incubator-devlake/issues/7851#issuecomment-2470051076
0.21 is outdated, please check the release page and update to the latest beta version .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Github] Pull Requests not being updated [incubator-devlake]
Posted by "ajain4-insight (via GitHub)" <gi...@apache.org>.
ajain4-insight commented on issue #7851:
URL: https://github.com/apache/incubator-devlake/issues/7851#issuecomment-2468337188
Hi, i have a similar issue, is this fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org