[jira] [Commented] (ACCUMULO-1755) BatchWriter blocks all addMutation calls while binning mutations

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173962#comment-15173962 ] 

ASF GitHub Bot commented on ACCUMULO-1755:
------------------------------------------

Github user keith-turner commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/75#discussion_r54590327
  
    --- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java ---
    @@ -699,7 +710,33 @@ else if (Tables.getTableState(context.getInstance(), table) == TableState.OFFLIN
     
         }
     
    -    void addMutations(MutationSet mutationsToSend) {
    +    void queueMutations(final MutationSet mutationsToSend) throws InterruptedException {
    +      if (null == mutationsToSend)
    +        return;
    +      boolean transferred = queue.tryTransfer(new Runnable() {
    --- End diff --
    
    I was playing around with this locally.  I added some prints and noticed the background thread was never binning.  This was because the background thread was never started (adding vial thread pool API will start it).  I fixed this issue by prestarting the threads, however  I think it would be safer to add using the Executor API (because threads could idle time out).  I was reading and found the following.
    
     * ThreadPoolExecutor javadocs recommend only using the queue for debugging and monitoring
     * The same thing can be accomplished using ThreadPoolExecutor.CallerRunsPolicy() and a SynchronousQueue()
    
    I got .CallerRunsPolicy+SynchronousQueue working here keith-turner/accumulo@bb28195
    
    I made some modifications to BatchWriterFlushIT in keith-turner/accumulo@3d3c252 .  Running that I was able to see that sometimes the background thread binned and sometimes the foreground thread binned.



> BatchWriter blocks all addMutation calls while binning mutations
> ----------------------------------------------------------------
>
>                 Key: ACCUMULO-1755
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1755
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Adam Fuchs
>            Assignee: Dave Marion
>             Fix For: 1.8.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Through code inspection, we found that the BatchWriter bins mutations inside of a synchronized block that covers calls to addMutation. Binning potentially involves lookups of tablet metadata and processes a fair amount of information. We will get better parallelism if we can either unlock the lock while binning, dedicate another thread to do the binning, or use one of the send threads to do the binning.
> This has not been verified empirically yet, so there is not yet any profiling info to indicate the level of improvement that we should expect. Profiling and repeatable demonstration of this performance bottleneck should be the first step on this ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)