[jira] [Created] (ACCUMULO-3797) Tablet not major compacting for >30minutes

["Josh Elser (JIRA)" <ji...@apache.org>]

Josh Elser created ACCUMULO-3797:
------------------------------------

             Summary: Tablet not major compacting for >30minutes
                 Key: ACCUMULO-3797
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3797
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
            Reporter: Josh Elser
             Fix For: 1.8.0, 1.7.1


Noticed a case during RandomWalk where scans against a table were blocked for a long time. There were too many files for the tablet, and batch scans were blocked.

Eventually, the major compaction _did_ start, but I'm not sure why it didn't start in order seconds to minutes. Eventually, it was noticed that the trace table (1) had a ton of WALs and needed a minor compaction. Shortly after that, a major compaction for our table (h6) was started.

{noformat}
2015-05-11 12:02:17,159 [tserver.TabletServer] DEBUG: Initiating minor compaction for 1<< because it has 35 write ahead logs
2015-05-11 12:02:17,160 [tserver.NativeMap] DEBUG: Allocated native map 0x000000000e198b30
2015-05-11 12:02:17,160 [tablet.Tablet] DEBUG: MinC initiate lock 0.00 secs
2015-05-11 12:02:17,160 [zookeeper.DistributedWorkQueue] DEBUG: Looking for work in /accumulo/374feac3-96ff-4ba1-99e6-8c1ab5f9d7c2/replication/workqueue
2015-05-11 12:02:17,172 [tablet.Tablet] DEBUG: Major compaction plan: [hdfs://cn020:8020/accumulo/tables/h6/t-0000hz8/C0000i04.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzh/C0000i0b.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hz
e/C0000i05.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzj/C0000i10.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hz3/C0000i0c.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzm/C0000i0x.rf, hdfs://cn020:8020/accumulo/tables/h6/t-00
00hzk/C0000i0v.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hza/C0000i0j.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzc/C0000i0f.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzi/C0000i0q.rf] propogate deletes : true
2015-05-11 12:02:17,172 [tablet.Tablet] DEBUG: MajC initiate lock 0.00 secs, wait 0.00 secs
2015-05-11 12:02:17,173 [tablet.Tablet] DEBUG: Starting MajC h6<< (NORMAL) [hdfs://cn020:8020/accumulo/tables/h6/t-0000hz3/C0000i0c.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hze/C0000i05.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0
000hzm/C0000i0x.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hza/C0000i0j.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzi/C0000i0q.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzh/C0000i0b.rf, hdfs://cn020:8020/accumulo/tables/h6
/t-0000hz8/C0000i04.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzk/C0000i0v.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzj/C0000i10.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzc/C0000i0f.rf] --> hdfs://cn020:8020/accumulo/t
ables/h6/default_tablet/C0000i2j.rf_tmp  []
2015-05-11 12:02:17,219 [tablet.MinorCompactor] DEBUG: Begin minor compaction hdfs://cn020:8020/accumulo/tables/1/default_tablet/F0000i2i.rf_tmp 1<<
2015-05-11 12:02:17,290 [tabletserver.LargestFirstMemoryManager] DEBUG: BEFORE compactionThreshold = 0.851 maxObserved = 11,134,533
2015-05-11 12:02:17,290 [tabletserver.LargestFirstMemoryManager] DEBUG: AFTER compactionThreshold = 0.851
2015-05-11 12:02:17,337 [tserver.TabletServer] DEBUG: ScanSess tid 172.18.128.20:35527 +rep 0 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
2015-05-11 12:02:17,833 [tablet.Compactor] DEBUG: Compaction h6<< 54,721 read | 54,721 written | 238,956 entries/sec |  0.229 secs
2015-05-11 12:02:17,841 [tserver.TabletServer] DEBUG: ScanSess tid 172.18.128.20:37393 +r 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
2015-05-11 12:02:17,841 [tserver.TabletServer] INFO : Adding 1 logs for extent !0<< as alias 230
2015-05-11 12:02:17,842 [tserver.TabletServer] INFO : Adding 1 logs for extent +r<< as alias 1
2015-05-11 12:02:17,941 [tablet.Compactor] DEBUG: Compaction 1<< 47,890 read | 47,713 written | 164,006 entries/sec |  0.292 secs
{noformat}

After the major compactions completed, things got back to normal. It seems highly unusual to sit there blocked for so long. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)