You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "dosubot[bot] (via GitHub)" <gi...@apache.org> on 2026/04/29 21:20:17 UTC
Re: [D] Question regarding scaling options. [incubator-devlake]
GitHub user dosubot[bot] edited a comment on the discussion: Question regarding scaling options.
<!-- Greeting -->
Hi @DucTuLeAnh! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the incubator-devlake team.
<!-- Answer -->
You're correct — DevLake is designed as a [single-instance application](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/devops/deployment/k8s/k8s-deploy.yaml) and doesn't support horizontal scaling by increasing replicas. All blueprints are [processed sequentially by a single runner](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317), which becomes the primary bottleneck at enterprise scale.
For 1000+ GitLab repositories, there have been community reports of significant challenges:
- [Full collection cycles taking ~2 weeks](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317) for 80+ projects
- [MySQL hitting 100% CPU at ~1,000 projects](https://github.com/apache/incubator-devlake/issues/8260) even with 16 cores / 64 GB RAM
- [Raw tables growing to 21+ GB](https://github.com/apache/incubator-devlake/issues/8523) with 2000 repos due to user data duplication
**What helps at scale:**
1. **Database indexes** — Community-tested indexes [reduced DORA calculations from hours to minutes](https://github.com/apache/incubator-devlake/issues/8260)
2. **GitLab-specific optimization** — Set `GITLAB_SERVER_COLLECT_ALL_USERS=false` to collect members per-project instead of globally ([reduced collection time by ~70%](https://github.com/apache/incubator-devlake/issues/8283))
3. **Memory allocation** — [Minimum 6 GB container memory](https://github.com/apache/incubator-devlake/discussions/8576#discussioncomment-14401772) recommended
4. **Collection strategy** — Use incremental collection, narrow date ranges, and consider `SKIP_COMMIT_STAT=true` / `SKIP_COMMIT_FILES=true`
The honest answer is that at 1000+ repositories, you'll likely face performance challenges even with optimizations. Parallel runners are the [most requested architectural enhancement](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317) for this reason.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fincubator-devlake).*
---
Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fincubator-devlake).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fincubator-devlake#message_id=88d0ad01-730f-4d10-a428-a3fe23565acf) [](https://github.dosu.com/apache/incubator-devlake?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fincubator-devlake) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fincubator-devlake)
GitHub link: https://github.com/apache/incubator-devlake/discussions/8802#discussioncomment-16276817
----
This is an automatically sent email for dev@devlake.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@devlake.apache.org