You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "renflo (via GitHub)" <gi...@apache.org> on 2023/08/10 10:29:59 UTC
[GitHub] [incubator-devlake] renflo opened a new issue, #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
renflo opened a new issue, #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
# Some context:
1. I am using devlake at work primarily collect DORA metrics and have therefore configured tens of devlake projects.
2. Some of the repositories scanned by the devlake projects have thousands of jobs.
3. To keep data fresh each devlake project is scheduled to run daily.
1. I've struggled a bit with parallelism my deployment is on k8s and I have scaled down to 1 lake pod to minimize them. But they are not entirely gone.
# The issue
The biggest issue I have right now is that some projects fail to gather data from gitlab, which means that even if the following steps in the devlake pipeline do succeed the resulting data is erroneous/mangled. The screen under documents how such a failed pipeline looks like.
That's very problematic because it happens relatively often and greatly diminishes the value of our DORA statistics.
# Available information
The "Task failed" popups (which I hope you can[ improve](https://github.com/apache/incubator-devlake/issues/5847)) show that some kind of error is occurring.
## Slice index out of range
The first one a bout the slice index out of range does not make any sense to me - but if sounds like a plain bug. I can find it in the logs, formatted in a more friendly way:
Hope you can look into it but I don't think it's the root cause of my issue - the next one is.
## Too Many Requests
The second error message is much more interesting. I find it in the logs as well and it seems pretty obvious that gitlab is not accepting the number of requests devlake sends to it. Do note that the following mentions a specific job: this job does exist in gitlab, so it has nothing to do with its existence.
```
stack trace
-- stack trace:
| github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks
| /app/core/runner/run_task.go:274
| [...repeated from below...]
Wraps: (2) subtask collectApiJobs ended unexpectedly
Wraps: (3) attached stack trace
-- stack trace:
|
github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
| /app/helpers/pluginhelper/api/worker_scheduler.go:173
|
github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
| /app/helpers/pluginhelper/api/api_collector.go:197
|
github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*ApiCollectorStateManager).Execute
| /app/helpers/pluginhelper/api/api_collector_with_state.go:112
|
github.com/apache/incubator-devlake/plugins/gitlab/tasks.CollectApiJobs
| /app/plugins/gitlab/tasks/job_collector.go:128
| github.com/apache/incubator-devlake/core/runner.runSubtask
| /app/core/runner/run_task.go:331
| github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks
| /app/core/runner/run_task.go:272
| github.com/apache/incubator-devlake/core/runner.RunPluginTask
| /app/core/runner/run_task.go:158
| github.com/apache/incubator-devlake/core/runner.RunTask
| /app/core/runner/run_task.go:134
|
github.com/apache/incubator-devlake/server/services.runTaskStandalone
| /app/server/services/task_runner.go:131
|
github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1
| /app/server/services/task.go:186
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1598
Wraps: (4)
| combined messages:
| {
| Retry exceeded 9 times calling projects/26623118/jobs/3134745242.
The last error was: Http DoAsync error calling [GET
projects/26623118/jobs/3134745242]. Response: <!DOCTYPE html>
| <html>
| <head>
| <meta content="width=device-width, initial-scale=1,
maximum-scale=1" name="viewport">
| <title>429 Too Many Requests</title>
| <style>
| body {
| color: #666;
| text-align: center;
| font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
| margin: auto;
| font-size: 14px;
| }
|
| h1 {
| font-size: 56px;
| line-height: 100px;
| font-weight: normal;
| color: #456;
| }
|
| h2 {
| font-size: 24px;
| color: #666;
| line-height: 1.5em;
| }
|
| h3 {
| color: #456;
| font-size: 20px;
| font-weight: normal;
| line-height: 28px;
| }
|
| hr {
| max-width: 800px;
| margin: 18px auto;
| border: 0;
| border-top: 1px solid #EEE;
| border-bottom: 1px solid white;
| }
|
| img {
| max-width: 40vw;
| }
|
| .container {
| margin: auto 20px;
| }
| </style>
| </head>
|
| <body>
| <h1>
| <img
src="data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRo
IGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo="
alt="GitLab Logo" /><br />
| 429
| </h1>
| <div class="container">
| <h3>Too many requests received.</h3>
| <hr />
| <p>Please try again later.</p>
| </div>
| </body>
| </html>
| (429)
| =====================
| Retry exceeded 9 times calling projects/26623118/jobs/3134745244.
The last error was: Http DoAsync error calling [GET
[...]
```
### What do you expect to happen
I expect the pipeline to successfully fetch data from gitlab - regardless of the size of the repository, number of other concurrent pipelines or throttling mecanisms in place at gitlab.
### How to reproduce
I cannot share the code I'm scanning.
But for you to reproduce I would imagine that scanning a repository with a large enough (thousands, hundreds? I must say that I don't know) number of already run jobs should do it.
### Anything else
Throttling and limiting the rate of API request to SaaS systems is common - devlake will likely offer many integrations that have throttling / rate limiting mechanism in place. In my case it's gitlab but it could be anything. Devlake needs to offer a solution to this.
Observations that could perhaps lead to a solution:
1. I suspect that devlake fetches ALL data from gitlab each time the pipeline runs (based on the fact that they seem to always take very long time, even from day to day). If it's the case an obvious optimization would be to only fetch data which has happened AFTER the latest collected data found in devlake db.
2. I am not sure what happens when I receive 429 Too Many Errors: some data has always been gathered before the error occurs. Has it been stored? I suspect that it isn't, and if it's discarded it really is a shame. Some data would be better than no data. In addition combined with 1 that would guarantee that a pipeline failing because of 429 would anyways gradually collect more and more data, and would at some point "catch up" and be able to run without failing.
### Version
0.18.0-beat4
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674191121
@renflo
1. Incremental collection support is listed via https://devlake.apache.org/docs/Overview/SupportedDataSources , I would like to avoid full collection, but it is highly dependent on the API capabilities, e.g. API have to support filtering records by time or sorting by it, etc.
2. Seems like you were running multiple pipelines in parallel to collect data from the same Gitlab server, it would certainly exceed the rate limit of the server since Rate Limit control is under the scope of a single pipeline at this point. We haven't figured out an elegant way to handle the rate limit over multiple concurrent pipelines yet, and most importantly, collecting the same GitLab server in parallel would not speed things up due to the Rate Limit, I think a better solution is to make them run in Sequential Order, and of course, it not available at this point, I suggest you set different times for each of them for the time being.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
URL: https://github.com/apache/incubator-devlake/issues/5849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1942889858
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1955470918
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1677068107
@renflo I'm afraid it is not that simple.
Pagination has nothing to do with the Incremental Sync, in fact, it is creating problems for the purpose. Keep in mind jobs are listed in Reverse Order, and not only do we have to fetch the new jobs, but also to update the existing records. Consider the following flow:
1. On the 1st day, we collect 2 pages, 10 records in total from the data source. 9 of them are succeeded or failed while one of them is **RUNNING**, all good.
2. On the 2nd day, 3 new records have been added, they appear on First page at 0-2 indexes.
So, what devlake is doing:
1. Fetch the First Page, extract the new records, and stop fetching the next page.
2. Iterate all Unfinished Jobs in the database and try to fetch them again to cover what was left from the previous step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] renflo commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "renflo (via GitHub)" <gi...@apache.org>.
renflo commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674235622
> Hi @renflo , were these ten projects collecting data at the same time of day?
@klesh : i have spread the pipelines over many hours a few months back, because as i mentioned concurrency issues has been a recurring topic. That does help a bit but not all jobs are taking the same amount of time so it's not trivial to spread them correcly - but it's doable (ui support is not great though, i need to click myself in in each project, i've thought of a feature request for that but it can wait).
Until concurrency is solved a "run all pipelines for this connection" sequentially would be great.
@Startrekzky no, actually i had failed pipelines with just two concurrent pipelines. Also i can add that by running one ine at a time even the slowest ones (ie largest most active repos). So that is also a temporary workaround for me but I really hope you don't close this issue because of it: the usability of the system is really difficult when i have to do so much fiddling and manual operations (eg spread batch job execution by hand,verify pipeline status by hand in each pipeline and the rerun them one by one until they all are green).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] renflo commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "renflo (via GitHub)" <gi...@apache.org>.
renflo commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674231377
@klesh i totally understand that you depend on feature support for "intervals" to be able to do things incrementally. I are you knowing for a fact that gitlab's api does not support that, i haven't checked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] d4x1 commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674181693
The "Slice index out of range" is easy to fix and I can help fix it.
The "Too Many Requests" is rate limited by gitlab, @klesh Do you have any advice about how to handle it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] renflo commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "renflo (via GitHub)" <gi...@apache.org>.
renflo commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674263355
@klesh the [link you gave](https://devlake.apache.org/docs/Overview/SupportedDataSources/) says that "Full Refresh, Incremental Sync(for issues,MRs)". So I guess we're discussing adding jobs to issues and MR ;)
I did some quick research and found the following out:
1. Gitlab supports pagination for the jobs endpoint https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-jobs
2.Gitlab support pages https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-jobs
3. devlake is using it at https://github.com/apache/incubator-devlake/blob/3469c1e786cc058dfcdeb4cd29765f21b8663738/backend/plugins/gitlab/tasks/job_collector.go#L71
So you already have something in place :)
I have not fine read the code so i'm not sure how you rely on gitlab's page feature. There are likely many ways of doing it to avoid refeshing everything all the time, for example this:
1. when the job collection task starts it selects the number of previously existing job info objects (haven't looked up what you store) in devlake's database. Say it's N.
2. `(N / per_page)+1` would give the page to which to start the listing from, which would be provided in `page` when querying gitlab
This ofc supposes the `per_page` value has not changed between runs. If that's a stretch (eg you haven't hard coded it as I suppose) you could likely store what `per_page` value was used at pipeline level and the current configured value at connection level. When any gitlab collection tasks starts that supports pages it would check the curremt configured value at connection level and compare it to value used during last successful pipeline run. If it's the same the algorithm other could be used. if it's not a full refresh would be necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5849: [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674221575
Hi @renflo , were these ten projects collecting data at the same time of day?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1857062616
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Lake] Fetching jobs from gitlab leads to 429 Too Many Requests [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5849:
URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1762415621
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org