You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "jackylau (Jira)" <ji...@apache.org> on 2019/10/11 10:04:00 UTC
[jira] [Commented] (HBASE-23149) hbase shouldPerformMajorCompaction
logic is not correct
[ https://issues.apache.org/jira/browse/HBASE-23149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949336#comment-16949336 ]
jackylau commented on HBASE-23149:
----------------------------------
hi [~zhangduo], it fould there is a bug in hbase1.9. could i commit a pr?
> hbase shouldPerformMajorCompaction logic is not correct
> -------------------------------------------------------
>
> Key: HBASE-23149
> URL: https://issues.apache.org/jira/browse/HBASE-23149
> Project: HBase
> Issue Type: Bug
> Components: Compaction
> Affects Versions: 1.4.9
> Reporter: jackylau
> Priority: Major
> Fix For: 1.4.11
>
>
> we can know that the major compaction is skipping from the below regionserver's log, but it is compacting that region.
> and read the code and find it is not correct and i add mark "/*** ***/" below
>
> public boolean shouldPerformMajorCompaction(final Collection<StoreFile> filesToCompact)
> throws IOException {
> if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
> if (filesToCompact.size() == 1) {
> if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
> float blockLocalityIndex =
> sf.getHDFSBlockDistribution().getBlockLocalityIndex(
> RSRpcServices.getHostname(comConf.conf, false));
> if (blockLocalityIndex < comConf.getMinLocalityToForceCompact())
> { result = true; }
> else
> { LOG.debug("Skipping major compaction of " + regionInfo + " because one (major) compacted file only, oldestTime " + oldest + "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex + " (min " + comConf.getMinLocalityToForceCompact() + ")"); }
> } else if (cfTTL != HConstants.FOREVER && oldest > cfTTL)
> { LOG.debug("Major compaction triggered on store " + regionInfo + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; }
> } else
> { LOG.debug("Major compaction triggered on store " + regionInfo + "; time since last major compaction " + (now - lowTimestamp) + "ms"); }
> result = true; /******here it is not right, it should be move to the above *****/
> }
> return result;
> }
>
> 2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.RatioBasedCompactionPolicy: Skipping major compaction of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. because one (major) compacted file only, oldestTime 3758085589ms is < TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
> 2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0 compacting, 1 eligible, 100 blocking
> 2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major compaction (all files)
> 2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.CompactSplitThread: Large Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1; Because: CompactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0, merge_queue=0
> 2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HRegion: Starting compaction on d in region 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
> 2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HStore: Starting compaction of 1 file(s) in d of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. into tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp, totalSize=5.1 G
> 2019-09-27 09:09:35,961 DEBUG [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] compactions.Compactor: Compacting hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6, keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF, compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371
--
This message was sent by Atlassian Jira
(v8.3.4#803005)