[jira] [Commented] (ACCUMULO-3797) Tablet not major compacting for >30minutes

["Josh Elser (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984144#comment-14984144 ] 

Josh Elser commented on ACCUMULO-3797:
--------------------------------------

[~ecn] [~phrocker], didn't you all observe this and make a fix for something similar? Might this be a duplicate issue now?

> Tablet not major compacting for >30minutes
> ------------------------------------------
>
>                 Key: ACCUMULO-3797
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3797
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>            Reporter: Josh Elser
>              Labels: 1.7.0_QA
>             Fix For: 1.7.1, 1.8.0
>
>
> Noticed a case during RandomWalk where scans against a table were blocked for a long time. There were too many files for the tablet, and batch scans were blocked.
> Eventually, the major compaction _did_ start, but I'm not sure why it didn't start in order seconds to minutes. Eventually, it was noticed that the trace table (1) had a ton of WALs and needed a minor compaction. Shortly after that, a major compaction for our table (h6) was started.
> {noformat}
> 2015-05-11 12:02:17,159 [tserver.TabletServer] DEBUG: Initiating minor compaction for 1<< because it has 35 write ahead logs
> 2015-05-11 12:02:17,160 [tserver.NativeMap] DEBUG: Allocated native map 0x000000000e198b30
> 2015-05-11 12:02:17,160 [tablet.Tablet] DEBUG: MinC initiate lock 0.00 secs
> 2015-05-11 12:02:17,160 [zookeeper.DistributedWorkQueue] DEBUG: Looking for work in /accumulo/374feac3-96ff-4ba1-99e6-8c1ab5f9d7c2/replication/workqueue
> 2015-05-11 12:02:17,172 [tablet.Tablet] DEBUG: Major compaction plan: [hdfs://cn020:8020/accumulo/tables/h6/t-0000hz8/C0000i04.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzh/C0000i0b.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hz
> e/C0000i05.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzj/C0000i10.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hz3/C0000i0c.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzm/C0000i0x.rf, hdfs://cn020:8020/accumulo/tables/h6/t-00
> 00hzk/C0000i0v.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hza/C0000i0j.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzc/C0000i0f.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzi/C0000i0q.rf] propogate deletes : true
> 2015-05-11 12:02:17,172 [tablet.Tablet] DEBUG: MajC initiate lock 0.00 secs, wait 0.00 secs
> 2015-05-11 12:02:17,173 [tablet.Tablet] DEBUG: Starting MajC h6<< (NORMAL) [hdfs://cn020:8020/accumulo/tables/h6/t-0000hz3/C0000i0c.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hze/C0000i05.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0
> 000hzm/C0000i0x.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hza/C0000i0j.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzi/C0000i0q.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzh/C0000i0b.rf, hdfs://cn020:8020/accumulo/tables/h6
> /t-0000hz8/C0000i04.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzk/C0000i0v.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzj/C0000i10.rf, hdfs://cn020:8020/accumulo/tables/h6/t-0000hzc/C0000i0f.rf] --> hdfs://cn020:8020/accumulo/t
> ables/h6/default_tablet/C0000i2j.rf_tmp  []
> 2015-05-11 12:02:17,219 [tablet.MinorCompactor] DEBUG: Begin minor compaction hdfs://cn020:8020/accumulo/tables/1/default_tablet/F0000i2i.rf_tmp 1<<
> 2015-05-11 12:02:17,290 [tabletserver.LargestFirstMemoryManager] DEBUG: BEFORE compactionThreshold = 0.851 maxObserved = 11,134,533
> 2015-05-11 12:02:17,290 [tabletserver.LargestFirstMemoryManager] DEBUG: AFTER compactionThreshold = 0.851
> 2015-05-11 12:02:17,337 [tserver.TabletServer] DEBUG: ScanSess tid 172.18.128.20:35527 +rep 0 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 2015-05-11 12:02:17,833 [tablet.Compactor] DEBUG: Compaction h6<< 54,721 read | 54,721 written | 238,956 entries/sec |  0.229 secs
> 2015-05-11 12:02:17,841 [tserver.TabletServer] DEBUG: ScanSess tid 172.18.128.20:37393 +r 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 2015-05-11 12:02:17,841 [tserver.TabletServer] INFO : Adding 1 logs for extent !0<< as alias 230
> 2015-05-11 12:02:17,842 [tserver.TabletServer] INFO : Adding 1 logs for extent +r<< as alias 1
> 2015-05-11 12:02:17,941 [tablet.Compactor] DEBUG: Compaction 1<< 47,890 read | 47,713 written | 164,006 entries/sec |  0.292 secs
> {noformat}
> After the major compactions completed, things got back to normal. It seems highly unusual to sit there blocked for so long. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)