You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2025/11/17 06:53:00 UTC
[jira] [Comment Edited] (HDDS-13933) Consistent Read from OM Followers
[ https://issues.apache.org/jira/browse/HDDS-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18038778#comment-18038778 ]
Ivan Andika edited comment on HDDS-13933 at 11/17/25 6:52 AM:
--------------------------------------------------------------
One difficulty is that
{code:java}
AlignmentContext#isCoordinatedCall(String protocolName, String method){code}
is not compatible with OzoneManagerService, since OzoneManagerService only contains a single RPC method (see HDDS-881 for the reason which seems to be for unifying Ratis conversion logic)
{code:java}
/**
The OM service that takes care of Ozone namespace.
*/
service OzoneManagerService {
// A client-to-OM RPC to send client requests to OM Ratis server
rpc submitRequest(OMRequest)
returns(OMResponse);
}{code}
HDFS, on the other hand, ClientNamenodeProtocol contains a single RPC method for each distinct RPC call
{code:java}
service ClientNamenodeProtocol {
rpc getBlockLocations(GetBlockLocationsRequestProto)
returns(GetBlockLocationsResponseProto);
rpc getServerDefaults(GetServerDefaultsRequestProto)
returns(GetServerDefaultsResponseProto);
rpc create(CreateRequestProto)returns(CreateResponseProto);
rpc append(AppendRequestProto) returns(AppendResponseProto);
... {code}
This means that for Ozone, the method will always be "submitRequest". This means that the ReadOnly annotation in HDFS cannot be used for Ozone.
The protocolName and method is taken from RequestHeaderProto which only contains those two fields, so we might need to extend it. However, since we need to take a look at the parameter OMRequest.cmdType , we might need to change the Hadoop RPC to also parse the OMRequest payload.
was (Author: JIRAUSER298977):
One difficulty is that
{code:java}
AlignmentContext#isCoordinatedCall(String protocolName, String method){code}
is not compatible with OzoneManagerService, since OzoneManagerService only contains a single RPC method (see HDDS-881 for the reason which is for Ratis compatibility)
{code:java}
/**
The OM service that takes care of Ozone namespace.
*/
service OzoneManagerService {
// A client-to-OM RPC to send client requests to OM Ratis server
rpc submitRequest(OMRequest)
returns(OMResponse);
}{code}
HDFS, on the other hand, ClientNamenodeProtocol contains a single RPC method for each distinct RPC call
{code:java}
service ClientNamenodeProtocol {
rpc getBlockLocations(GetBlockLocationsRequestProto)
returns(GetBlockLocationsResponseProto);
rpc getServerDefaults(GetServerDefaultsRequestProto)
returns(GetServerDefaultsResponseProto);
rpc create(CreateRequestProto)returns(CreateResponseProto);
rpc append(AppendRequestProto) returns(AppendResponseProto);
... {code}
This means that for Ozone, the method will always be "submitRequest". This means that the ReadOnly annotation in HDFS cannot be used for Ozone.
The protocolName and method is taken from RequestHeaderProto which only contains those two fields, so we might need to extend it. However, since we need to take a look at the parameter OMRequest.cmdType , we might need to change the Hadoop RPC to also parse the OMRequest payload.
> Consistent Read from OM Followers
> ---------------------------------
>
> Key: HDDS-13933
> URL: https://issues.apache.org/jira/browse/HDDS-13933
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> HDDS-9279 introduces the Ratis / Raft based follower reads by giving an option to enable Ratis Linearizable Read and Leader Lease feature. However, based on previous performance tests, there are significant performance regressions both in the client and server which makes it not production ready. Currently, the root cause of this performance hit has yet to be discovered.
> Therefore, I suggest we pursue two concurrent approaches for consistent follower read. We can then compare the two approach and enable the one with the better performance.
> * Improve the Ratis Lineariable Read and ReadIndex improvements (following up on HDDS-9279)
> ** In my opinion, since we are stuck on improving this, we might try the second approach
> * (This ticket) We follow the HDFS observer read implementation (HDFS-12943)
> ** General flow
> *** Client send an msync to OM leader and OM leader reply with the last applied index as part of the AlignmentContext#getLastSeenId
> *** When client send a request to the OM follower, the Hadoop RPC mechanism will detect the AlignmentContext and will requeue the call if the Call#getClientStateId() > AlignmentContext.getLastSeenStateId() (see Hadoop Server.Handler#run)
> **** This implies that the server RPC handler will not get blocked, unlike linearizable read
> ** Pros
> *** There is already a reference implementation on HDFS, we can simply finesse it to Ozone context
> **** e.g. FSImage#getLastAppliedOrWrittenTxId can be translated to StateMachine#getLastAppliedTermIndex
> *** This is similar to HDFS observer read implementation, so we know that this level of consistency is acceptable and we don't need a lot to prove that it is correct (or at least acceptable)
> **** If we know that HDFS observer read has been deployed to production with no consistency issue and with acceptable performance so we expect the same on Ozone
> *** Possible better performance since Server will requeue the call to the RPC server instead of blocking
> ** Cons
> *** Only supports Hadoop RPC based client (gRPC based client is not supported and requires its own development)
> *** Will drift the implementation away from Raft / Ratis
> ** Current Implementation plans
> *** Ozone side
> **** AlignmentContext for client
> ***** Implementation is similar to Hadoop ClientGSIContext
> **** AlignmentContext for OM
> ***** Implementation is similar to Hadoop GlobalStateIdContext
> **** Create a new proxy provider similar to ObserverReadProxyProvider to allow to route read requests to the OM followers
> *** Hadoop side
> **** Support parsing AlignmentContext for ProtobufRpcEngine (raised HADOOP-19741)
> ***** See Hadoop Server.Connection#processRequest
> Additionally I hope that implementing one can uncover issues and improvements on the other.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org