You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "klesh (via GitHub)" <gi...@apache.org> on 2024/04/26 04:15:54 UTC
[I] [Feature][jira] Jira changelog extractor and converter support Incremental Mode [incubator-devlake]
klesh opened a new issue, #7385:
URL: https://github.com/apache/incubator-devlake/issues/7385
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
### Use case
Reduce the time to crunching massive amounts of data
### Description
Currently, Extractors and Converters operate exclusively in Full Sync mode, which involves deleting all target data and regenerating it via a Delete + Insert process. While effective, this approach poses several issues:
1. **Scalability Concerns**: As the volume of records increases in the source tables, the time required for conversion scales linearly. In particular, operations such as the jira issue changelogs extraction and conversion have been reported to take up to half an hour. This is significantly slower than the data collection phase, impacting overall efficiency.
2. **Database Efficiency**: Running in Full Sync mode tends to cause database bloat, particularly in databases like PostgreSQL. This bloat is evidenced by the table size being disproportionately large compared to the actual data stored — in some cases, as extreme as 18GB of space used for 1GB of actual data.
#### Proposed Solution:
I propose that extractors and converters should be enhanced to support Incremental Mode. This mode would enable the components to only process and insert new or changed data since the last collection, rather than performing a full refresh each time. This would likely yield the following benefits:
1. **Reduced Processing Time**: Incremental updates would significantly reduce the time required for data conversion, as only new or changed records would be processed.
2. **Improved Database Performance**: By avoiding the deletion and re-insertion of large volumes of data, we can prevent database bloat, leading to better utilization of resources and potentially lower storage costs.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][jira] Jira changelog extractor and converter support Incremental Mode [incubator-devlake]
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky closed issue #7385: [Feature][jira] Jira changelog extractor and converter support Incremental Mode
URL: https://github.com/apache/incubator-devlake/issues/7385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org