You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "GoSimplicity (via GitHub)" <gi...@apache.org> on 2025/08/21 08:16:30 UTC
[PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
GoSimplicity opened a new pull request, #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547
Fixed the Epic collector to use the new Jira API endpoint and ensured all required fields are included when fetching Epic data. This resolves the synchronization failures caused by the deprecated endpoint (fix #8544 ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "GoSimplicity (via GitHub)" <gi...@apache.org>.
GoSimplicity commented on PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#issuecomment-3227240459
> I see that there are new commits, and the code looks good to me. Just to confirm, is it ready for merging? Have you tested it against Jira Cloud, and does the pagination work as expected?
Yes, the code is ready for merging. I have done some testing and it works properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "GoSimplicity (via GitHub)" <gi...@apache.org>.
GoSimplicity commented on PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#issuecomment-3218686665
> hanks for your contribution! This should resolve the JIRA Cloud data collection issue. However, I’m concerned it may not work with older JIRA Server versions. I think we should detect the JIRA version and call the appropriate APIs accordingly.
>
> For reference, here’s some related code:
>
> https://github.com/apache/incubator-devlake/blob/ac3b1d9cb317e6a08b1412ddef0de7bbb9b97c9e/backend/plugins/jira/tasks/epic_collector.go#L54
Based on your suggestion, I have made the new and old version endpoints compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "GoSimplicity (via GitHub)" <gi...@apache.org>.
GoSimplicity commented on code in PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#discussion_r2297084047
##########
backend/plugins/jira/tasks/epic_collector.go:
##########
@@ -98,6 +117,10 @@ func CollectEpics(taskCtx plugin.SubTaskContext) errors.Error {
query.Set("startAt", fmt.Sprintf("%v", reqData.Pager.Skip))
Review Comment:
Excellent catch on the pagination changes! You're absolutely correct that api/3/search/jql uses nextPageToken instead of startAt. I'll update the implementation to:
1.Use different pagination strategies based on the API version
2.Set concurrency to 1 for api/3 (sequential pagination only)
3.Handle the nextPageToken properly in the response parser
I'll revise the PR to address these issues. Would you prefer separate collector implementations for each API version, or a single collector with conditional logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#issuecomment-3227157857
I see that there are new commits, and the code looks good to me. Just to confirm, is it ready for merging? Have you tested it against Jira Cloud, and does the pagination work as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#discussion_r2297351537
##########
backend/plugins/jira/tasks/epic_collector.go:
##########
@@ -98,6 +117,10 @@ func CollectEpics(taskCtx plugin.SubTaskContext) errors.Error {
query.Set("startAt", fmt.Sprintf("%v", reqData.Pager.Skip))
Review Comment:
There’s already a mechanism for handling this kind of API. Here’s how you can enable it:
1. Specify `GetNextPageCustomData` in `ApiCollectorArgs` to extract the `nextPageToken`.
- This token will be stored in the `RequestData.CustomData` field.
- Note: this also forces the collector to run in sequential mode (no parallel fetching).
2. In the `Query` function, read the `CustomData` and plug it into the query string.
Here’s an example:
https://github.com/apache/incubator-devlake/blob/52433a8bc098eba668a02d815d618409aa375b93/backend/plugins/bitbucket_server/tasks/pr_collector.go#L47-L48
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on code in PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#discussion_r2297073016
##########
backend/plugins/jira/tasks/epic_collector.go:
##########
@@ -98,6 +117,10 @@ func CollectEpics(taskCtx plugin.SubTaskContext) errors.Error {
query.Set("startAt", fmt.Sprintf("%v", reqData.Pager.Skip))
Review Comment:
Based on the information provided in the Atlassian developer changelog, the new `api/3/search/jql` endpoint has changed the way data is paginated. Specifically, random page access has been replaced with a continuation token API. This means that you won't be able to fetch multiple pages simultaneously using parallel threads. The `startAt` parameter has been replaced with `nextPageToken`.
Given these changes, simply updating the API endpoint without adjusting the pagination logic will result in incomplete data retrieval. The existing code that relies on the `startAt` parameter for pagination will no longer work as expected. It's crucial to implement the new nextPageToken mechanism to handle pagination properly and ensure that all data can be fetched correctly.
> We’ll replace random page access with a continuation token API. This means you won’t be able to get multiple pages at the same time with parallel threads. startAt parameter will be replaced with nextPageToken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "GoSimplicity (via GitHub)" <gi...@apache.org>.
GoSimplicity commented on PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547#issuecomment-3218776775
> That was fast — great job!
>
> However, I made a mistake in pointing to an inappropriate line of code for this case. The version check on line 54 was originally added because the performance of Jira versions ≤ 8 was quite poor, so we reduced the batch size to avoid overloading or crashing the Jira server.
>
> In the current scenario, checking only the server type should be sufficient. I'm not sure whether Jira Server 9 supports the `api/v3` endpoint or not.
You're right about the performance considerations for JIRA Server ≤ 8. I'll keep the reduced batch size for those versions. Regarding JIRA Server 9's support for api/v3, I'll need to investigate further and potentially add version-specific handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(jira): update epic collector to use new API endpoint and include [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh merged PR #8547:
URL: https://github.com/apache/incubator-devlake/pull/8547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org