[jira] [Commented] (HDDS-16435) OM LINEARIZABLE_ALLOW_FOLLOWER read can return stale data from a superseded leader

["Ivan Andika (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115904#comment-18115904 ] 

Ivan Andika commented on HDDS-16435:
------------------------------------

This is an already known issue, leadership check for each follower read would kill the already bad follower read performance.

> OM LINEARIZABLE_ALLOW_FOLLOWER read can return stale data from a superseded leader
> ----------------------------------------------------------------------------------
>
>                 Key: HDDS-16435
>                 URL: https://issues.apache.org/jira/browse/HDDS-16435
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Ivan Andika
>            Priority: Major
>         Attachments: HDDS-16435.001.patch
>
>
> h3. Finding
> Under the opt-in leader-skip config (LINEARIZABLE_ALLOW_FOLLOWER), a superseded leader still self-reporting LEADER_AND_READY serves a raw local read with no ReadIndex or leadership recheck, so a client that explicitly asked for no-stale reads can silently receive committed data older than an acknowledged write during a partition-induced split-brain window. Three related model-checking findings share this root cause: MC-1 (the RPC read path), MC-2 (LINEARIZABLE_ALLOW_FOLLOWER routed to a stale-capable raw local read while LINEARIZABLE_LEADER_ONLY keeps ReadIndex), and CR-2 (on leader fall-back the gRPC/S3 gateway transport re-hints to DEFAULT while the RPC transport keeps LINEARIZABLE_ALLOW_FOLLOWER). Bounded, externally observable linearizability (safety) violation via a missing leadership/quorum check. ENV_LIMITED: MiniOzoneHACluster cannot inject the asymmetric partition, so the routing/hint divergence was reproduced but the live stale value was not.
> h3. Classification
> * Verdict: ENV_LIMITED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1/MC-2/CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula:      v1.1.0 (commit c6aa3dfa)
> Target:       om-follower-read
> Guidance:     campaigns/ozone-9fbf9ee/targets/001-om-follower-read/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
>   --enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
>   "om-follower-read|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16428 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Deploy a 3 node OM HA Raft group with linearizable read enabled (ozone.om.read.option=LINEARIZABLE) and set ozone.om.allow.leader.skip.linearizable.read=true.
> # A client issues a read carrying the LINEARIZABLE_ALLOW_FOLLOWER consistency hint. The proto documents this level as ensuring there are no stale reads, and when follower read is enabled it is also the client default read consistency.
> # The request lands on an OM that still believes it is the leader: getLeaderStatus returns LEADER_AND_READY, a value derived purely from the local Ratis role without any quorum confirmation.
> # Because the leader skip flag is set, submitReadRequestToOmLinearizableAllowFollower serves the request with a raw local read (handler.handleReadRequest) and skips the ReadIndex barrier.
> # If that OM has been superseded by a new leader during a partition (its term is behind the new leader's term), its local committed state is behind the cluster, so the client that explicitly asked for no stale reads receives stale data.
> Unmasking condition: the stale value is only observed when the serving OM is a superseded leader that still reports LEADER_AND_READY. That requires an asymmetric network partition isolating the old leader from the quorum while it stays reachable to clients. A current leader returns the fresh value, so the routing divergence is masked in a healthy cluster, and the partition cannot be injected in the unit or MiniOzoneHACluster test harness. The routing defect itself is deterministic; only the live stale read is environment limited.
> h3. Root cause
> The read routing in OzoneManagerProtocolServerSideTranslatorPB treats the two linearizable levels inconsistently on a ready leader. submitReadRequestToOmLinearizableLeaderOnly always uses the ReadIndex path on LEADER_AND_READY, but submitReadRequestToOmLinearizableAllowFollower first checks isAllowLeaderSkipLinearizableRead and, when it is set, returns a raw local read with no ReadIndex confirmation. getLeaderStatus (OzoneManagerRatisServer) reflects only the node's local Ratis role, so a superseded leader still passes the LEADER_AND_READY guard for a window. As a result an explicit LINEARIZABLE_ALLOW_FOLLOWER request is silently downgraded to the DEFAULT serve local behavior, defeating the documented no stale reads guarantee and diverging from LINEARIZABLE_LEADER_ONLY. The leader skip flag is a performance optimization meant for the no hint path (where DEFAULT already permits a stale read window during a partition), and its config description discloses no consistency impact on the explicit linearizable levels.
> A related transport gap: on follower read fall back the RPC failover proxy keeps the LINEARIZABLE_ALLOW_FOLLOWER hint while the gRPC transport re hints the leader retry to DEFAULT, so the two transports can disagree on the effective consistency of the same fall back read. This shares the same underlying hazard (an explicit linearizable follower read being downgraded to a stale capable leader local read) and is noted here for the reviewer; the attached patch addresses the server side leader skip downgrade, which is the primary and common path.
> h3. Suggested fix
> Do not apply the leader skip optimization to an explicit LINEARIZABLE_ALLOW_FOLLOWER read. The attached patch [^HDDS-16435.001.patch] removes the leader skip branch from submitReadRequestToOmLinearizableAllowFollower so that on a ready leader the request always takes the ReadIndex path, matching LINEARIZABLE_LEADER_ONLY. The optimization stays active for the no hint (DEFAULT) read path, whose contract already allows a stale read window.
> The added unit test (TestOmReadConsistencyRouting) drives the read routing through processRequest with the leader skip flag enabled and asserts that a LINEARIZABLE_ALLOW_FOLLOWER read on a LEADER_AND_READY node takes the ReadIndex path and never the leader skip local read, and that both linearizable levels route identically. Both assertions fail on the current code and pass with the fix. A full end to end reproduction of the stale value requires a superseded leader window from an asymmetric partition, which the in process test harness cannot inject.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full test, and review before merge.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org