You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "moulivashisth (via GitHub)" <gi...@apache.org> on 2025/09/12 11:50:17 UTC
[PR] implemented keyset pagination for gitlab #8529 [incubator-devlake]
moulivashisth opened a new pull request, #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
### ⚠️ Pre Checklist
> Please complete _ALL_ items in this checklist, and remove before submitting
- [ x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).
- [ x] I have added relevant tests.
- [ x] I have added relevant documentation.
- [ x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.
<!--
Thanks for submitting a pull request!
We appreciate you spending the time to work on these changes.
Please fill out as many sections below as possible.
-->
### Summary
**Fix GitLab Users collection hitting offset pagination limits by adding keyset pagination.**
This PR updates the GitLab `CollectAccounts` subtask to avoid `max offset` errors when collecting large user sets:
- **Self-managed GitLab instances** now use **keyset pagination** on `/api/v4/users`
(`pagination=keyset&order_by=id&sort=asc&per_page=N&id_after=<last_id>`) and **do not** send `page`.
- **gitlab.com / jihulab.com** keep existing behavior on project member endpoints
(`/projects/:id/members[/all]`) which typically remain under offset caps per project.
- Retains existing API-version fallback (`/members/all` vs `/members/` for < v13.11).
- Response parser now tracks the last item’s `id` to advance the keyset cursor safely.
- No breaking changes to task wiring or raw table schema (`gitlab_api_users`).
**Why:** Some instances enforce strict offset caps (e.g., 50k), causing `offset pagination is restricted` errors when fetching Users. Keyset pagination removes the offset and enables full retrieval.
**Risk/Compatibility:**
- Backward compatible; only changes query parameters and cursor handling.
- If a project’s members list exceeds offset caps and the endpoint lacks keyset, users should collect site users (self-managed path) or shard by project/group—unchanged from current guidance.
### Does this close any open issues?
Closes 8529 ([Bug][GitLab] Pagination not working Again)
### Screenshots
N/A
### Other Information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gitlab): implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3865850728
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3301795900
Is `keyset` and `api/v4` available in `Community Edition 11+` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gitlab): implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8572: feat(gitlab): implemented keyset pagination for gitlab #8529
URL: https://github.com/apache/incubator-devlake/pull/8572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gitlab): implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3829874556
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3326379834
Hi, you may run the following command to fix the golangci-lint error:
```
gofmt -s -w -l plugins/gitlab/tasks/account_collector.go
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "moulivashisth (via GitHub)" <gi...@apache.org>.
moulivashisth commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3305459364
Thanks for bringing this up!
• Keyset pagination for /api/v4/users was introduced in 16.5, and 17.0+ requires it for large responses.
I’ve updated the code to gate keyset by server version (≥16.5) and fallback to offset otherwise. Project members endpoints keep their existing offset behavior.
This preserves compatibility with CE 11–16.4 while avoiding offset-cap failures on newer instances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] implemented keyset pagination for gitlab #8529 [incubator-devlake]
Posted by "moulivashisth (via GitHub)" <gi...@apache.org>.
moulivashisth commented on PR #8572:
URL: https://github.com/apache/incubator-devlake/pull/8572#issuecomment-3316839187
Thanks for the review.
I hit a typecheck error:
`plugins/gitlab/tasks/account_collector.go:69:55: undefined: apiVersion`
Fix: Declared and reused a single apiVersion variable sourced from the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org