[jira] [Commented] (KUDU-1078) Under heavy load, log cache reads return "Op in future"

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314410#comment-15314410 ] 

Todd Lipcon commented on KUDU-1078:
-----------------------------------

Looking at this again in the context of KUDU-1469 (which might be entirely unrelated, but made me look into this code path).

I think the explanation is:
- in LogCache::AppendOperations, we do the following order:
1. under lock: add new operations to the cache map
2. outside lock: Log::AsyncAppendReplicates (which may block when under load)
3. under lock: increase next_sequential_op_index_

- in LogCache::ReadOps(...), we do:
1. Look for the operation in the cache.
2. If it's a hit, iterate forward in the cache until we stop "hitting"

This, ReadOps() can return those operations that are currently in-flight on their way into the log, even before AsyncAppendReplicates completes. That's not in and of itself bad -- the point of the log cache is to provide exactly this "write through caching" so that we can replicate operations remotely before they've been written locally and ride over leader log latency blips. However, we can get the following interleaving which causes the error messages:

*Thread 1*
- call LogCache::AppendOperations() with batch 10..20
-- inserts 10..20 into the cache
-- blocks on AsyncAppendReplicates

*Thread 2*
- preparing a request for a follower, calls ReadOps(10)
-- results in ops 10..20 being copied into the outgoing request
- sends the request

* Later, while Thread 1 is still blocked *
- response received from the peer for 10..20. 'last_replicated' = 20, so next_index = 21
- we call ReadOps(21)
-- next_sequential_op_id_ has still not advanced from 10
-- this triggers the error message indicated in this JIRA

I think, then, this error is probably harmless, though it's a signature of a blocked leader log.



> Under heavy load, log cache reads return "Op in future"
> -------------------------------------------------------
>
>                 Key: KUDU-1078
>                 URL: https://issues.apache.org/jira/browse/KUDU-1078
>             Project: Kudu
>          Issue Type: Improvement
>          Components: consensus
>    Affects Versions: Private Beta
>            Reporter: Todd Lipcon
>            Assignee: Mike Percy
>
> JD accidentally configured botl80 so that all nodes were writing to a single tablet (talk about a stress test!) I see the following warning occasionally in the leader logs:
> W0827 11:34:04.051275 55950 consensus_queue.cc:272] T d6ff74fb04454712873abd0d2328e59b P 668314723deb4a818cc9a43eba51073c [LEADER]: The logs necessary to catch up peer 600e62435ca24425b29c00d9726b78be have been garbage collected. The follower will never be able to catch up (Not found: op in future). Instructing remote peer to remotely bootstrap.
> Calling this a blocker because, if we couple this with the "automatically re-bootstrap nodes if they fall behind", we'll be accidentally deleting tablets when this happens (no good!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)