You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Simon Zhang (JIRA)" <ji...@apache.org> on 2019/02/15 11:44:00 UTC
[jira] [Commented] (KUDU-1078) Under heavy load, log cache reads
return "Op in future"
[ https://issues.apache.org/jira/browse/KUDU-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769223#comment-16769223 ]
Simon Zhang commented on KUDU-1078:
-----------------------------------
Hi [~tlipcon], I use kudu-1.8.0, seems still have this issue:
{code:java}
E0130 16:55:58.582803 9979 consensus_queue.cc:699] T bbaf907dfbf04c24b46fff873a94237f P cbc554095f1f4ef5b6442da45a542ac3 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 731 is ahead of the local log (next sequential op: 731). Destination peer: Peer: permanent_uuid: "a33504d2e2fc4447aa054f2589b9f9ae" member_type: VOTER last_known_addr { host: "cm07" port: 7050 }, Status: INVALID_TERM, Last received: 1645.730, Next index: 732, Last known committed idx: 731, Time since last communication: 1.356s
E0130 16:55:58.583446 8244 consensus_queue.cc:699] T 23e82946c5704607bbe118674c493c62 P cbc554095f1f4ef5b6442da45a542ac3 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 726 is ahead of the local log (next sequential op: 726). Destination peer: Peer: permanent_uuid: "083ef4d758854ddd9f4d15a3c718fe4b" member_type: VOTER last_known_addr { host: "cm04" port: 7050 }, Status: INVALID_TERM, Last received: 1623.726, Next index: 727, Last known committed idx: 726, Time since last communication: 1.358s
E0130 16:55:58.755020 5500 consensus_queue.cc:699] T b5ec089a30fb4b24a353ebbd6d58e7f6 P cbc554095f1f4ef5b6442da45a542ac3 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 740 is ahead of the local log (next sequential op: 740). Destination peer: Peer: permanent_uuid: "083ef4d758854ddd9f4d15a3c718fe4b" member_type: VOTER last_known_addr { host: "cm04" port: 7050 }, Status: INVALID_TERM, Last received: 1670.740, Next index: 741, Last known committed idx: 740, Time since last communication: 1.570s
E0130 16:55:58.756345 5723 consensus_queue.cc:699] T dc3790caa17b40a79b0d94cb9aa8e69a P cbc554095f1f4ef5b6442da45a542ac3 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 751 is ahead of the local log (next sequential op: 751). Destination peer: Peer: permanent_uuid: "083ef4d758854ddd9f4d15a3c718fe4b" member_type: VOTER last_known_addr { host: "cm04" port: 7050 }, Status: INVALID_TERM, Last received: 1613.751, Next index: 752, Last known committed idx: 751, Time since last communication: 1.584s
{code}
can you help to give some help?
> Under heavy load, log cache reads return "Op in future"
> -------------------------------------------------------
>
> Key: KUDU-1078
> URL: https://issues.apache.org/jira/browse/KUDU-1078
> Project: Kudu
> Issue Type: Improvement
> Components: consensus
> Affects Versions: Private Beta
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Fix For: 1.6.0
>
>
> JD accidentally configured botl80 so that all nodes were writing to a single tablet (talk about a stress test!) I see the following warning occasionally in the leader logs:
> W0827 11:34:04.051275 55950 consensus_queue.cc:272] T d6ff74fb04454712873abd0d2328e59b P 668314723deb4a818cc9a43eba51073c [LEADER]: The logs necessary to catch up peer 600e62435ca24425b29c00d9726b78be have been garbage collected. The follower will never be able to catch up (Not found: op in future). Instructing remote peer to remotely bootstrap.
> Calling this a blocker because, if we couple this with the "automatically re-bootstrap nodes if they fall behind", we'll be accidentally deleting tablets when this happens (no good!)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)