You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "squatboy (via GitHub)" <gi...@apache.org> on 2026/03/31 08:54:07 UTC

[I] [Bug][Github GraphQL] Deleted or transferred issues leave stale local rows after #8637 [incubator-devlake]

squatboy opened a new issue, #8819:
URL: https://github.com/apache/incubator-devlake/issues/8819

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found the related transferred-issue fix #7969 / PR #8637.
   
   ### What happened
   
   GitHub GraphQL issue refresh no longer fails immediately on `Could not resolve to an Issue` after #8637, but deleted or transferred issues can still leave stale local rows in DevLake.
   
   Those stale rows remain eligible for future refresh attempts because the local `_tool_github_issues` and raw rows are not cleaned up when the source issue no longer resolves.
   
   In the reproduced case, a previously collected GitHub issue was deleted from the source repository. DevLake had already collected it into local tables, and later refresh attempts still tried to resolve it through GitHub GraphQL.
   
   ### What do you expect to happen
   
   When GitHub GraphQL returns `Could not resolve to an Issue` for an issue in the refresh-open-issues path, DevLake should:
   
   1. continue processing the rest of the batch
   2. remove or tombstone the stale local issue rows that keep the missing issue in the refresh path
   3. avoid retrying the same orphaned issue forever
   
   ### How to reproduce
   
   1. Create a GitHub issue in repository A
   2. Run a DevLake pipeline that collects GitHub issues for repository A
   3. Delete the GitHub issue from repository A, or transfer it to another repository
   4. Re-run the DevLake pipeline
   5. Observe that DevLake still has stale local rows for the old issue and continues to encounter the missing issue in the refresh path
   
   ### Anything else
   
   Related upstream fix:
   
   - #7969
   - #8637
   
   The remaining gap seems to be stale local data cleanup rather than just skipping the GraphQL error.
   
   ### Version
   
   Reproduced on `v1.0.3-beta9@1410838`
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's Code of Conduct
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][Github GraphQL] Deleted or transferred issues leave stale local rows after #8637 [incubator-devlake]

Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8819:
URL: https://github.com/apache/incubator-devlake/issues/8819#issuecomment-4161070743

   <!-- Greeting -->
   Hi @squatboy! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the incubator-devlake team.
   
   <!-- Answer -->
   This is a valid gap. After digging into the code, here's what's happening and a couple of additional things worth noting for whoever picks up the fix:
   
   **Current behavior after PR #8637:** The `fetchAsync()` in [`graphql_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L276-L284) now catches `&quot;Could not resolve to an Issue&quot;` errors and logs a warning instead of crashing the task — but the stale row in `_tool_github_issues` is never cleaned up. The Phase 2 refresh path queries all local issues with `state='OPEN'` [[1]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go#L149-L161), so the orphaned issue keeps appearing in every future refresh cycle indefinitely.
   
   **The `_tool_github_issues` model has no tombstone or soft-delete fields** — no `deleted_at`, `is_stale`, etc. [[2]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github/models/issue.go), so a fix would need to either hard-delete the row, update `State` to `CLOSED` (or a new sentinel value), or add a new field.
   
   **Batch-collateral issue:** There's a subtle additional problem — after the `dataErrors` loop in `fetchAsync()`, there's an early `return` [[3]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L281). This means if *one* issue in a batch of 100 triggers `&quot;Could not resolve&quot;`, the `ResponseParser` is **skipped for the entire batch**, so valid issues in the same batch also miss their refresh update for that cycle.
   
   A fix would likely need to address at least two things:
   
   1. **Stale row cleanup** — when the error is detected, delete or mark the corresponding `_tool_github_issues` row so it's excluded from future Phase 2 refresh queries.
   2. **Batch handling** — avoid the early `return` so that valid issues in the same batch still get processed by `ResponseParser`. One approach could be to set `IgnoreQueryErrors: true` (like [`account_collector.go` and `job_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L82) already do) and handle the error inside the `ResponseParser` instead, or to filter out the errored entries and continue processing the rest of the batch.
   
   Since you indicated you're willing to submit a PR — the key files to look at are [`backend/plugins/github_graphql/tasks/issue_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go) (Phase 2 refresh logic) and [`backend/helpers/pluginhelper/api/graphql_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go) (the `fetchAsync` error handling).
   
   
   <!-- Dosu Comment Footer -->
   *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=brand-link).*
   
   ---
   <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=great_response&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=irrelevant_answer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=incorrect_sources&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=too_verbose&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e
 -4cbb-8856-8e54f9e7ef56?feedback_type=hallucination&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=bug_report&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-bug_report) | [Other](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=other&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-other)</sup>
   
   [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=chat-badge)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20is%20a%20valid%20gap.%20After%20digging%20into%20the%20code%2C%20here%27s%20what%27s%20happening%20and%20a%20couple%20of%20additional%20things%20worth%20noting%20for%20whoever%20picks%20up%20the%20fix%3A%0A%0A%2A%2ACurrent%20behavior%20after%20PR%20%238637%3A%2A%2A%20The%20%60fetchAsync%28%29%60%20in%20%5B%60graphql_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L276-L284%29%20now%20catches%20%60%26quot%3BCould%20not%20resolve%20to%20an%20Issue%26quot%3B%60%20errors%20and%20logs%20a%20warning%20instead%20of%20crashing%20the%20task%20%E2%80%94%20bu
 t%20the%20stale%20row%20in%20%60_tool_github_issues%60%20is%20never%20cleaned%20up.%20The%20Phase%202%20refresh%20path%20queries%20all%20local%20issues%20with%20%60state%3D%27OPEN%27%60%20%5B%5B1%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go%23L149-L161%29%2C%20so%20the%20orphaned%20issue%20keeps%20appearing%20in%20every%20future%20refresh%20cycle%20indefinitely.%0A%0A%2A%2AThe%20%60_tool_github_issues%60%20model%20has%20no%20tombstone%20or%20soft-delete%20fields%2A%2A%20%E2%80%94%20no%20%60deleted_at%60%2C%20%60is_stale%60%2C%20etc.%20%5B%5B2%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github/models/issue.go%29%2C%20so%20a%20fix%20would%20need%20to%20either%20hard-delete%20the%20row%2C%20update%20%60State%60%20to%20%60CLOSED%60%20%28or%20a%20new%20sentinel%20value%29%2C%20or%20add%20a%20new%20field.%0A%
 0A%2A%2ABatch-collateral%20issue%3A%2A%2A%20There%27s%20a%20subtle%20additional%20problem%20%E2%80%94%20after%20the%20%60dataErrors%60%20loop%20in%20%60fetchAsync%28%29%60%2C%20there%27s%20an%20early%20%60return%60%20%5B%5B3%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L281%29.%20This%20means%20if%20%2Aone%2A%20issue%20in%20a%20batch%20of%20100%20triggers%20%60%26quot%3BCould%20not%20resolve%26quot%3B%60%2C%20the%20%60ResponseParser%60%20is%20%2A%2Askipped%20for%20the%20entire%20batch%2A%2A%2C%20so%20valid%20issues%20in%20the%20same%20batch%20also%20miss%20their%20refresh%20update%20for%20that%20cycle.%0A%0AA%20fix%20would%20likely%20need%20to%20address%20at%20least%20two%20things%3A%0A%0A1.%20%2A%2AStale%20row%20cleanup%2A%2A%20%E2%80%94%20when%20the%20error%20is%20detected%2C%20delete%20or%20mark%20the%20corresponding%20%60_tool_github_issues%60%20row%20so%20it%27s%20excl
 uded%20from%20future%20Phase%202%20refresh%20queries.%0A2.%20%2A%2ABatch%20handling%2A%2A%20%E2%80%94%20avoid%20the%20early%20%60return%60%20so%20that%20valid%20issues%20in%20the%20same%20batch%20still%20get%20processed%20by%20%60ResponseParser%60.%20One%20approach%20could%20be%20to%20set%20%60IgnoreQueryErrors%3A%20true%60%20%28like%20%5B%60account_collector.go%60%20and%20%60job_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L82%29%20already%20do%29%20and%20handle%20the%20error%20inside%20the%20%60ResponseParser%60%20instead%2C%20or%20to%20filter%20out%20the%20errored%20entries%20and%20continue%20processing%20the%20rest%20of%20the%20batch.%0A%0ASince%20you%20indicated%20you%27re%20willing%20to%20submit%20a%20PR%20%E2%80%94%20the%20key%20files%20to%20look%20at%20are%20%5B%60backend/plugins/github_graphql/tasks/issue_collector.go%60%5D%28https%3A//github.com/apach
 e/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go%29%20%28Phase%202%20refresh%20logic%29%20and%20%5B%60backend/helpers/pluginhelper/api/graphql_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%29%20%28the%20%60fetchAsync%60%20error%20handling%29.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=join-discord)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8819)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][Github GraphQL] Deleted or transferred issues leave stale local rows after #8637 [incubator-devlake]

Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh closed issue #8819: [Bug][Github GraphQL] Deleted or transferred issues leave stale local rows after #8637
URL: https://github.com/apache/incubator-devlake/issues/8819


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org