[GitHub] [iotdb] THUMarkLau opened a new pull request, #6324: [IOTDB-3189] Fix compaction is not well-distributed across sgs

[GitBox <gi...@apache.org>]

THUMarkLau opened a new pull request, #6324:
URL: https://github.com/apache/iotdb/pull/6324

   See [IOTDB-3189(https://issues.apache.org/jira/browse/IOTDB-3189).
   
   In present IoTDB compaction task submitted order set by the `ICompactionTaskComparator`, currently this interface only has `DefaultCompactionTaskComparator` an implementation. `DefaultCompactionTaskComparator` compares the compaction task priority with following several aspects:
   - Whether the compaction priority of the system is Balance. If not, the prior one is inner space task or cross space task.
   - For inner space compaction, compare whether the tasks are sequence, compaction task level,compaction files version number, total number of compaction files, and total size of compaction files
   - For cross space compaction, compare the total number of files and the total size of them.
   Different storage groups are not considered in this comparison. Therefore, when the tasks of different storage groups are compared side by side, it is easy to create uneven storage combinations. For example, if two SG groups commit a large number of compacton tasks in the sequence space with the same number of files, the same file size, and the same file level, the system may compact tasks in only one storage group due to the different submission timing.
   To balance compaction tasks between different storage groups, when all other things being equal, each compaction task can be assigned a unique serial id that is counted individually by each virtual storage group and monotonically increases from zero. If two compaction tasks are from different storage groups, their sequence numbers are compared; the compaction task with a smaller sequence number has a higher priority. After adjustment, compaction priority comparison becomes
   - Whether the compaction priority of the system is Balance. If not, the prior one is inner space task or cross space task.
   - For inner space compaction, compare whether the tasks are sequence, compaction task level,compaction files version number, total number of compaction files, the serial number of compaction tasks, and the total size of files
   - For a cross space compaction, compare the total number of compaction files, the sequence number of compaction tasks, and the total size of files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] qiaojialin merged pull request #6324: [IOTDB-3189] Fix compaction is not well-distributed across sgs

[GitBox <gi...@apache.org>]

qiaojialin merged PR #6324:
URL: https://github.com/apache/iotdb/pull/6324


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org