[GitHub] [hbase] nyl3532016 edited a comment on pull request #3425: HBASE-25991 Do compaction on compaction server

[GitBox <gi...@apache.org>]

nyl3532016 edited a comment on pull request #3425:
URL: https://github.com/apache/hbase/pull/3425#issuecomment-872368228


   > Now I have another question. Seems we maintain the compacting files in compaction server. Keep compacting files is useful if you want to run multiple compactions on the same store. But how do we make sure that we always send the compaction request of a store to the same compaction server? If we can not guarantee, then keeping compacting files in compaction server is useless...
   
   1.We keep the region to CS mapping in Master(CompactionOffloadManager#selectCompactionServer),Now just use the hash code of region start key mod the compactionserver size (_hashcode(region.startkey) % compactionServerList.size()_) as index to mapping the compaction sever, it works well. maybe later we need an assignment module for compaction offload.
   2.And this mapping is not strictly required, If compactionServer changed from CS1 to CS2. Now we select file on CS2 may select the file alreay selected on CS1, So one of the compaction job must be failed when report to RS(RS will check if select file whether exsit). We need to ensure that the cache(compacting files in CompactionServerStorage) will be cleared once compaction job failed or crash. So keeping compacting files in compaction server will simplify the situation of compactionServer crash.
   3.Furthermore, we have an idea, utilize compaction server as another level block cache between RS and hdfs, to mitigate the block cache miss caused by compaction(at the time new file replace old compacted files). It need we have region to CS mapping  to locate which CS has block cache for the region.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org