You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "narrowizard (via GitHub)" <gi...@apache.org> on 2025/08/04 01:09:19 UTC
[I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
narrowizard opened a new issue, #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
We integrated a GitLab data source with over 2000 repositories into Apache DevLake. After running the data collection pipeline for a period, we observed that the `_raw_gitlab_api_users` table has grown to an unexpectedly large size, reaching approximately 21GB. Upon querying the table, we found a significant number of duplicate records for user data. This suggests that the raw layer data cleaning process for GitLab API users might not be functioning effectively.
### What do you expect to happen
We expect the `_raw_gitlab_api_users` table to contain unique user records without excessive duplicates. The table size should be reasonable and reflect the actual number of unique users, not inflated by redundant entries. The raw layer data cleaning mechanism should correctly deduplicate user data ingested from GitLab.
### How to reproduce
1. Integrate a GitLab data source in Apache DevLake.
2. Configure the data source to collect data from a large number of repositories (e.g., 2000+).
3. Run the data collection pipeline for an extended period (e.g., several days or weeks).
4. Monitor the size of the `_raw_gitlab_api_users` table in your database.
5. Query the `_raw_gitlab_api_users` table to identify and count duplicate user records.
### Anything else
_No response_
### Version
main
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523#issuecomment-3148856824
<!-- Answer -->
This is a known issue: the GitLab plugin collects user data on a per-project basis, so when you integrate many repositories, the `_raw_gitlab_api_users` table accumulates a large number of duplicate records and can grow very large. This behavior is expected given the current design—raw tables are meant to store all occurrences for traceability, and deduplication only happens later in the tool layer, not in the raw layer. There is an open discussion and a proposed solution to collect user accounts only once per pipeline, which would reduce duplication and table size, but it hasn't been merged yet. You can follow the details and progress in [issue #8283](https://github.com/apache/incubator-devlake/issues/8283). For now, periodic cleanup or manual deduplication may be needed if storage is a concern.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/9f462dbc-36a1-4ea7-938b-28a48a2361e4?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=githu
b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8523)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard closed issue #8523: [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration
URL: https://github.com/apache/incubator-devlake/issues/8523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523#issuecomment-3162307562
<!-- Answer -->
Your suspicion is correct: the plugin uses different logic for GitLab Server and Cloud. For self-hosted GitLab Server, it collects all users globally via the `/users` endpoint for every repository (data scope), which causes the same user data to be ingested multiple times and leads to excessive duplication in the `_raw_gitlab_api_users` table. For GitLab Cloud (gitlab.com, jihulab.com), it collects users per project, so duplication is less severe but still possible if users belong to multiple projects. This behavior is determined by a simple string match on the API endpoint in the code, which is brittle and can lead to inconsistent results if more SaaS domains are introduced or endpoint URLs change [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/gitlab/tasks/account_collector.go).
For diagnostics, I suggest enabling debug logging during collection and capturing the actual API responses and parameters for both environments. Compare the number and content of user records collected from Server vs. Cloud to confirm the duplication pattern. Also, check if the same user IDs appear multiple times for different repositories in the raw table.
As an interim mitigation, you can periodically deduplicate or clean up the `_raw_gitlab_api_users` table with a scheduled script, or limit the frequency of full user data collection. For large-scale integrations, consider adapting the workaround proposed in [issue #8283](https://github.com/apache/incubator-devlake/issues/8283) to collect users only once per pipeline for GitLab Server, which has been shown to significantly reduce table size and duplication.
If you test with different GitLab Server versions or configurations, sharing your findings will help refine a long-term solution for the community. No merged PR addresses this yet, so feedback and testing on proposed workarounds are valuable.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/a41da11d-20e9-4498-9765-5a48d84c27cc?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=githu
b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8523)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on issue #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523#issuecomment-3162300063
After testing with the cloud version of GitLab (using jihulab.com for testing), we did not observe the issue of excessive data volume and duplicate records in the `_raw_gitlab_api_users` table. This leads us to suspect that the problem might be specific to the GitLab Server version. We will conduct further tests in different environments and scenarios to identify the root cause and find a more effective solution. Thank you for your attention to this issue. We will keep you updated on our testing progress and findings.
@dosu do you have any further suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on issue #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523#issuecomment-3162777917
Disscussion before: https://github.com/apache/incubator-devlake/pull/6424#discussion_r1413300210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Gitlab] Excessive Data Size and Duplicates in `_raw_gitlab_api_users` Table After GitLab Integration [incubator-devlake]
Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on issue #8523:
URL: https://github.com/apache/incubator-devlake/issues/8523#issuecomment-3162644089
After further testing, we have confirmed that the situation with GitLab Server also aligns with the current design. In a GitLab Server instance with over 3000 users, integrating more than 2000 repositories results in each user record being duplicated over 2000 times in the `_raw_gitlab_api_users` table. This leads to a significant increase in data volume.
This finding highlights the existing issue in our current design, where the raw layer does not perform deduplication, causing the data volume to balloon due to the excessive number of duplicate records. This not only puts pressure on storage but also complicates subsequent data processing tasks.
We need to continue exploring more effective solutions to address this issue and improve our data collection and processing efficiency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org