You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "caioq (via GitHub)" <gi...@apache.org> on 2024/11/19 17:02:21 UTC
[I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
caioq opened a new issue, #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
The problem occurs during subsequent executions of a blueprint after the initial or full-refresh data collection has been completed successfully. Initially, all data is collected perfectly, and there are no errors or unusual behavior logged during these runs. However, in some cases, during subsequent runs of the same blueprint, old issues seem to be deleted, leaving only issues that were created or updated after the last pipeline execution.
### What do you expect to happen
Records should be incrementally added to the `issues` table.
### How to reproduce
This started happening 2 weeks ago in the production environment. I don't know if the [JIRA incident on November 5-6](https://jira-software.status.atlassian.com/) may have impacted this collection process.
### Anything else
Upon further investigation, I found that the `_raw_jira_api_issues` table still contains 73,145 records, while the `_tool_jira_issues` and `issues` tables only hold 1,115 records. This suggests there might be an issue with the extractor or data transformation during these runs.
### Version
v1.0.1-beta8@e195107 or v1.0.2-beta4
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2486272257
<!-- Answer -->
I found a similar closed issue that might be related to your problem: [Bug][jira] Jira issue extractor produces records less than expected](https://github.com/apache/incubator-devlake/issues/4709). This issue was closed, but it might provide some insights into the behavior you're experiencing <sup>[[1]](https://github.com/apache/incubator-devlake/issues/4709)</sup>.
Regarding your specific issue, it seems like the incremental data update process in Apache DevLake's Jira plugin might be affecting the records in the `issues` table. The process involves using a `StatefulApiExtractor` that, when in incremental mode, deletes existing records related to the issue in the `JiraIssueLabel` and `JiraIssueRelationship` tables before processing new data. This is intended to ensure that any changes to labels or relationships are accurately reflected in the database <sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/jira/tasks/issue_extractor.go)</sup>.
Given the discrepancy between the `_raw_jira_api_issues` and the other tables, it might be worth investigating whether the deletion process is affecting more records than intended. Additionally, consider checking if there are any specific configurations or settings that might be influencing this behavior.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8198: [Bug][Jira] Jira Issues Data Collection is deleting old registers
URL: https://github.com/apache/incubator-devlake/issues/8198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2526832728
I wonder if this issue is related to https://github.com/apache/incubator-devlake/issues/7826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2644364916
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2660561795
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2490198733
Any new project was created recently?
Was there any parallel pipeline running at the same time?
Is there any shared Jira board between projects?
Try executing the "Retransform Data" and see if the issues reappear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][Jira] Jira Issues Data Collection is deleting old registers [incubator-devlake]
Posted by "caioq (via GitHub)" <gi...@apache.org>.
caioq commented on issue #8198:
URL: https://github.com/apache/incubator-devlake/issues/8198#issuecomment-2514323874
> Any new project was created recently? Was there any parallel pipeline running at the same time? Is there any shared Jira board between projects?
>
> Try executing the "Retransform Data" and see if the issues reappear.
Hi @klesh !
When this problem arose, no project/configuration was created or changed.
Fortunately, when I retransform the data, the issues reappear 🙏🏼
Unfortunately, this past weekend this problem happened again. It's not something that happens often. I'm still investigating what could have happened differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org