[jira] [Commented] (ACCUMULO-4066) Conditional mutation processing performance could be improved.

["Josh Elser (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120803#comment-15120803 ] 

Josh Elser commented on ACCUMULO-4066:
--------------------------------------

This one slipped in under my radar. Thought I'd give your changes a glance. 3x speed up is awesome!

{code}
-    for (Condition cond : cm.getConditions()) {
+    // sort conditions inorder to get better lookup performance. Sort on client side so tserver does not have to do it.
+    Condition[] ca = cm.getConditions().toArray(new Condition[cm.getConditions().size()]);
+    Arrays.sort(ca, CONDITION_COMPARATOR);
{code}

To confirm, the server doesnt' rely on the sorted order, just hopes for it for performance reasons?

I see a lot of changes in IteratorUtil (I assume to your point about loading iterators from the table config). How did this used to work? You had lots of new tests added for the other cases -- do we have good coverage for IteratorUtil already?

> Conditional mutation processing performance could be improved.
> --------------------------------------------------------------
>
>                 Key: ACCUMULO-4066
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4066
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>    Affects Versions: 1.6.4, 1.7.0
>            Reporter: Keith Turner
>            Assignee: Keith Turner
>             Fix For: 1.6.5, 1.7.1, 1.8.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When processing conditional mutations tablets reads are done.   The way the current implementation does tablet reads has a lot of overhead.   For each condition the following is done :
>  * Opens and reserves iterators files.
>  * Parse table iterators from table config (involves scanning and filtering entire table config)
>  * Merges condition iterators and table iterators
>  * Constructs iterator stack.
> I created a branch where these operations (except for constructing iterator stack) are done per tablet and/or per batch of conditional mutations.   Doing this I am seeing a 3x speed up in conditional mutation processing rates when data is cached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)