You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "jackylau (Jira)" <ji...@apache.org> on 2019/10/11 10:02:00 UTC
[jira] [Created] (HBASE-23149) hbase shouldPerformMajorCompaction
logic is not correct
jackylau created HBASE-23149:
--------------------------------
Summary: hbase shouldPerformMajorCompaction logic is not correct
Key: HBASE-23149
URL: https://issues.apache.org/jira/browse/HBASE-23149
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 1.4.9
Reporter: jackylau
Fix For: 1.4.11
we can know that the major compaction is skipping but it is compacting that region from the below regionserver's log.
and read the code and find it is not correct and i add mark "/*** ***/" below
public boolean shouldPerformMajorCompaction(final Collection<StoreFile> filesToCompact)
throws IOException {
if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
if (filesToCompact.size() == 1) {
if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
float blockLocalityIndex =
sf.getHDFSBlockDistribution().getBlockLocalityIndex(
RSRpcServices.getHostname(comConf.conf, false));
if (blockLocalityIndex < comConf.getMinLocalityToForceCompact()) {
result = true;
} else {
LOG.debug("Skipping major compaction of " + regionInfo
+ " because one (major) compacted file only, oldestTime " + oldest
+ "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex
+ " (min " + comConf.getMinLocalityToForceCompact() + ")");
}
} else if (cfTTL != HConstants.FOREVER && oldest > cfTTL) {
LOG.debug("Major compaction triggered on store " + regionInfo
+ ", because keyvalues outdated; time since last major compaction "
+ (now - lowTimestamp) + "ms");
result = true;
}
} else {
LOG.debug("Major compaction triggered on store " + regionInfo
+ "; time since last major compaction " + (now - lowTimestamp) + "ms");
}
result = true; /******here it is not right, it should be move to the above *****/
}
return result;
}
2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.RatioBasedCompactionPolicy: Skipping major compaction of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. because one (major) compacted file only, oldestTime 3758085589ms is < TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0 compacting, 1 eligible, 100 blocking
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major compaction (all files)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.CompactSplitThread: Large Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1; Because: CompactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0, merge_queue=0
2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HRegion: Starting compaction on d in region 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HStore: Starting compaction of 1 file(s) in d of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. into tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp, totalSize=5.1 G
2019-09-27 09:09:35,961 DEBUG [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] compactions.Compactor: Compacting hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6, keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF, compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371
--
This message was sent by Atlassian Jira
(v8.3.4#803005)