[jira] [Commented] (ACCUMULO-1052) Minor compactions not finishing before master kills tabletserver can very large number of files per tablet

["Adam Fuchs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574769#comment-13574769 ] 

Adam Fuchs commented on ACCUMULO-1052:
--------------------------------------

Commits shouldn't be held just because a tablet is minor compacting, right? The tablet server has to run out of space in the write buffer across all tablets, or the tablets would have to be in a state of migration or going offline. Do you know what's causing the commit hold? Are minor compactions on tablets on this tablet server less efficient (this should be evident in the performance debug logs for that tablet server)? Could this be that the merging minor compactions are kicking in and reducing the compaction efficiency?

If the number of files is growing unbounded then that seems to indicated the merging minor compactions are not kicking in. Is that a separate issue, or perhaps something related? How many files are referenced by the tablet in question?
                
> Minor compactions not finishing before master kills tabletserver can very large number of files per tablet
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-1052
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1052
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: master, tserver
>    Affects Versions: 1.4.2
>         Environment: Large, write-heavy cluster
>            Reporter: Josh Elser
>            Assignee: Eric Newton
>
> On a cluster that is being saturated with heavy ingest, a tserver is observed attempting to perform a minor compaction for a tablet with multiple WALs. Because of this, commits to this tablet end up being held.
> After churning on the minc for some time, the master's hold-time limit for tservers is exceeded, however the minc didn't finish. The tserver is forcibly killed, the tablet is migrated, recovery occurs on the new tserver and the problem repeats.
> Some of the minor compactions must finish, as the number of files for that tablet continue to grow, but major compactions must not have time to finish since the number of files grow unbounded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira