You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ratis.apache.org by tison <wa...@gmail.com> on 2023/08/27 04:20:29 UTC
Read my write with async client
Hi,
I have asked a related question before[1] about syncing better query
and applytransaction.
Now, when I dive deeper into the Raft-based system, I notice a new case for
consistency.
Take the following executions as an example (supposed we implement a Map
statemachine):
Client 1 put k1 as v1
Client 1 get k1
Generally, we expect the get requests gets v1. But if these two requests
are issued in async, it's not always true.
Even the client TCP connection can guarantee that "get k1" goes after "put
k1", and we do read index for "get k1", the read index returned can be
lower then "put k1" if it's not yet committed. Then the "get k1" request
gets an empty value.
I'm not sure what is the best practices to implement read my write for a
Ratis-based system.
1. If we're using blocking IO, it won't be an issue cause "get k1" must be
after "put k1" committed and even applied.
2. If we build an extra facade over Ratis like IoTDB does, we can check the
read conditions as I did at [2].
3. But what if we only customize the statemachine and use the Ratis client?
I don't know which happens before relations I can depend on for concurrent
query and applytransaction.
Also, in [2] for raft-rs there is one more subtle issue that their
ReadIndex request can get lost or silently ignored, which causes the caller
to retry that can even get a much later read index. I'm curious if Ratis
has the same issue (lose ReadIndex or silently ignore and the caller
doesn't get response forever).
Best,
tison.
[1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
[2]
https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
Re: Read my write with async client
Posted by tison <wa...@gmail.com>.
Cross post that it's implemented at https://github.com/apache/ratis/pull/913
Best,
tison.
Tsz Wo Sze <sz...@gmail.com> 于2023年8月30日周三 02:03写道：
> > The issue of thenApply is the same as blocking IO that we cannot
> pipeline requests but only send the next read after the pioneer write
> return from the server.
>
> Async thenApply is better then blocking IO in the sense that thenApply
> only blocks the read requests requiring Read-After-Write Consistency,
> while blocking IO will block all the requests. Consider the following
>
> - Client 1: async put k1 as v1 and get future f1
> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
> - Client 1: async put k2 as v2
> - Client 1: async read k3
>
> The last two requests won't be blocked.
>
> > 1. Who drive the new read index process?
> > 2. How to fulfill "the previous request's index" as read index?
>
> The leader should drive the new read index process. When a read request
> requires Read-After-Write Consistency, the request will get the log index
> of the last write request of the same client and use it as the read index.
> It seems we need to add a CompletableFuture<long> logIndexFuture.
>
> The followers are irrelevant since they do not have support write.
>
> > I’ll fire a issue if we plan to implement it.
>
> Sure, let's do it. :)
>
> Tsz-Wo
>
> On Tue, Aug 29, 2023 at 12:16 AM William Song <sz...@163.com> wrote:
>
>> > It seems easy to modify the read-index algorithm to support it: use the
>> index of the previous request as the read index, instead of the commit
>> index.
>>
>> I’ll fire a issue if we plan to implement it.
>>
>> > We can resolve this issue if we use Ratis as IoTDB that wraps the
>> RaftServer with a facade and handles server-side connection customizedly.
>> That is, the client pipelines requests, but the server sequences the
>> requests and `submitClientRequest` with thenApply chain.
>>
>> IoTDB’s RaftServer facade is more like a customized client, rather than a
>> server. It pipelines the caller’s requests and then forwards them to the
>> Raft server sequentially.
>>
>> I think a `Connection` abstract inside which each request casually
>> depends on all the previously requests is what you are asking for. A
>> potential workaround is to implement a `Connection` which accepts async
>> call but use `thenApply` internally to arrange these async calls. It can
>> be officially implemented into Ratis if it’s a common demand.
>>
>> Hope it helps,
>> William
>>
>>
>>
>>
>> 2023年8月29日 07:14，tison <wa...@gmail.com> 写道：
>>
>> > use the index of the previous request as the read index, instead of
>> the commit index
>>
>> Yes. This is exactly what we semantically want to do. Here are two
>> questions we need to answer:
>>
>> 1. Who drive the new read index process?
>> 2. How to fulfill "the previous request's index" as read index?
>>
>> They can be the same problem.
>>
>> If the client drives the process, then the only way it knows "the
>> previous request's index" is waiting for the previous request return. This
>> is no better than blocking IO or thenApply.
>>
>> If the server drives the process, then "the previous request" means the
>> previous request of "this connection". I don't find a connection
>> abstraction in the codebase and this is what I encounter issues on the
>> first place: query/applyTransaction accept requests from all the clients
>> synchronously and not even in order to when they arrive the server.
>>
>> Best,
>> tison.
>>
>>
>> tison <wa...@gmail.com> 于2023年8月29日周二 07:00写道：
>>
>>> Thank Wiliam & Tzs Wo!
>>>
>>> Yeah I just realized that it's somehow a "client-side" or
>>> "connection-aspect" issue.
>>>
>>> The issue of thenApply is the same as blocking IO that we cannot
>>> pipeline requests but only send the next read after the pioneer write
>>> return from the server.
>>>
>>> We can resolve this issue if we use Ratis as IoTDB that wraps the
>>> RaftServer with a facade and handles server-side connection customizedly.
>>> That is, the client pipelines requests, but the server sequences the
>>> requests and `submitClientRequest` with thenApply chain. This is like how
>>> ZooKeeper's CommitProcessor chains its read/write requests[1].
>>>
>>> So here we may have another perspective on the question: what are the
>>> expected methods of application based on Ratis?
>>>
>>> If we want users to write applications with RaftClient in App's
>>> client-side and only customized StateMachine on App's server side, even use
>>> RaftServer as the server facade, then the current query/applyTransaction,
>>> and even the fixed one-shot RPC[2] can be a limitation.
>>>
>>> If we encourage users to wrap RaftServer with their own server facade
>>> and use RaftClient in the App's server-side on demand like IoTDB does, then
>>> maybe the experience of submitting requests on the server side can be
>>> improved; or we provide some examples to do so. (Generally, it should be
>>> fine to always use RaftClient in the App's server-side, while it can cause
>>> one extra codec, but we already have to send the request to other peers on
>>> writes.)
>>>
>>> Best,
>>> tison.
>>>
>>> [1]
>>> https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259
>>> [2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm
>>>
>>> Tsz Wo Sze <sz...@gmail.com> 于2023年8月29日周二 02:36写道：
>>>
>>>> HI tison, William,
>>>>
>>>> Thanks for pointing out the problem!
>>>>
>>>> For the long term, we should support Read-After-Write Consistency. It
>>>> seems easy to modify the read-index algorithm to support it: use the index
>>>> of the previous request as the read index, instead of the commit index.
>>>>
>>>> As a work around, we may use the async API with thenApply:
>>>> - Client 1: async put k1 as v1 and get future f1
>>>> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get
>>>> k1}).
>>>>
>>>> Tsz-Wo
>>>>
>>>>
>>>> On Mon, Aug 28, 2023 at 8:28 AM William Song <sz...@163.com>
>>>> wrote:
>>>>
>>>>> Hi tison,
>>>>>
>>>>> The readIndex provides only the linearizable consistency guarantee. A
>>>>> linearizable read returned by RaftServer at least reflects any changes
>>>>> already committed and replied to the client. If you require a more strict
>>>>> consistency guarantee, like read reflecting the latest write, you may need
>>>>> to add extra synchronization logic. I think blocking IO or dependencies on
>>>>> completionStage will both work.
>>>>>
>>>>> Regards,
>>>>> William
>>>>>
>>>>>
>>>>> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have asked a related question before[1] about syncing better query
>>>>> and applytransaction.
>>>>>
>>>>> Now, when I dive deeper into the Raft-based system, I notice a new
>>>>> case for consistency.
>>>>>
>>>>> Take the following executions as an example (supposed we implement a
>>>>> Map statemachine):
>>>>>
>>>>> Client 1 put k1 as v1
>>>>> Client 1 get k1
>>>>>
>>>>> Generally, we expect the get requests gets v1. But if these two
>>>>> requests are issued in async, it's not always true.
>>>>>
>>>>> Even the client TCP connection can guarantee that "get k1" goes after
>>>>> "put k1", and we do read index for "get k1", the read index returned can be
>>>>> lower then "put k1" if it's not yet committed. Then the "get k1" request
>>>>> gets an empty value.
>>>>>
>>>>> I'm not sure what is the best practices to implement read my write for
>>>>> a Ratis-based system.
>>>>>
>>>>> 1. If we're using blocking IO, it won't be an issue cause "get k1"
>>>>> must be after "put k1" committed and even applied.
>>>>> 2. If we build an extra facade over Ratis like IoTDB does, we can
>>>>> check the read conditions as I did at [2].
>>>>> 3. But what if we only customize the statemachine and use the Ratis
>>>>> client? I don't know which happens before relations I can depend on for
>>>>> concurrent query and applytransaction.
>>>>>
>>>>> Also, in [2] for raft-rs there is one more subtle issue that their
>>>>> ReadIndex request can get lost or silently ignored, which causes the caller
>>>>> to retry that can even get a much later read index. I'm curious if Ratis
>>>>> has the same issue (lose ReadIndex or silently ignore and the caller
>>>>> doesn't get response forever).
>>>>>
>>>>> Best,
>>>>> tison.
>>>>>
>>>>> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
>>>>> [2]
>>>>> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>>>>>
>>>>>
>>>>>
>>
Re: Read my write with async client
Posted by Tsz Wo Sze <sz...@gmail.com>.
> The issue of thenApply is the same as blocking IO that we cannot pipeline
requests but only send the next read after the pioneer write return from
the server.
Async thenApply is better then blocking IO in the sense that thenApply only
blocks the read requests requiring Read-After-Write Consistency, while blocking
IO will block all the requests. Consider the following
- Client 1: async put k1 as v1 and get future f1
- Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
- Client 1: async put k2 as v2
- Client 1: async read k3
The last two requests won't be blocked.
> 1. Who drive the new read index process?
> 2. How to fulfill "the previous request's index" as read index?
The leader should drive the new read index process. When a read request
requires Read-After-Write Consistency, the request will get the log index
of the last write request of the same client and use it as the read index.
It seems we need to add a CompletableFuture<long> logIndexFuture.
The followers are irrelevant since they do not have support write.
> I’ll fire a issue if we plan to implement it.
Sure, let's do it. :)
Tsz-Wo
On Tue, Aug 29, 2023 at 12:16 AM William Song <sz...@163.com> wrote:
> > It seems easy to modify the read-index algorithm to support it: use the
> index of the previous request as the read index, instead of the commit
> index.
>
> I’ll fire a issue if we plan to implement it.
>
> > We can resolve this issue if we use Ratis as IoTDB that wraps the
> RaftServer with a facade and handles server-side connection customizedly.
> That is, the client pipelines requests, but the server sequences the
> requests and `submitClientRequest` with thenApply chain.
>
> IoTDB’s RaftServer facade is more like a customized client, rather than a
> server. It pipelines the caller’s requests and then forwards them to the
> Raft server sequentially.
>
> I think a `Connection` abstract inside which each request casually depends
> on all the previously requests is what you are asking for. A potential
> workaround is to implement a `Connection` which accepts async call but use
> `thenApply` internally to arrange these async calls. It can be officially
> implemented into Ratis if it’s a common demand.
>
> Hope it helps,
> William
>
>
>
>
> 2023年8月29日 07:14，tison <wa...@gmail.com> 写道：
>
> > use the index of the previous request as the read index, instead of the
> commit index
>
> Yes. This is exactly what we semantically want to do. Here are two
> questions we need to answer:
>
> 1. Who drive the new read index process?
> 2. How to fulfill "the previous request's index" as read index?
>
> They can be the same problem.
>
> If the client drives the process, then the only way it knows "the previous
> request's index" is waiting for the previous request return. This is no
> better than blocking IO or thenApply.
>
> If the server drives the process, then "the previous request" means the
> previous request of "this connection". I don't find a connection
> abstraction in the codebase and this is what I encounter issues on the
> first place: query/applyTransaction accept requests from all the clients
> synchronously and not even in order to when they arrive the server.
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2023年8月29日周二 07:00写道：
>
>> Thank Wiliam & Tzs Wo!
>>
>> Yeah I just realized that it's somehow a "client-side" or
>> "connection-aspect" issue.
>>
>> The issue of thenApply is the same as blocking IO that we cannot pipeline
>> requests but only send the next read after the pioneer write return from
>> the server.
>>
>> We can resolve this issue if we use Ratis as IoTDB that wraps the
>> RaftServer with a facade and handles server-side connection customizedly.
>> That is, the client pipelines requests, but the server sequences the
>> requests and `submitClientRequest` with thenApply chain. This is like how
>> ZooKeeper's CommitProcessor chains its read/write requests[1].
>>
>> So here we may have another perspective on the question: what are the
>> expected methods of application based on Ratis?
>>
>> If we want users to write applications with RaftClient in App's
>> client-side and only customized StateMachine on App's server side, even use
>> RaftServer as the server facade, then the current query/applyTransaction,
>> and even the fixed one-shot RPC[2] can be a limitation.
>>
>> If we encourage users to wrap RaftServer with their own server facade and
>> use RaftClient in the App's server-side on demand like IoTDB does, then
>> maybe the experience of submitting requests on the server side can be
>> improved; or we provide some examples to do so. (Generally, it should be
>> fine to always use RaftClient in the App's server-side, while it can cause
>> one extra codec, but we already have to send the request to other peers on
>> writes.)
>>
>> Best,
>> tison.
>>
>> [1]
>> https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259
>> [2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm
>>
>> Tsz Wo Sze <sz...@gmail.com> 于2023年8月29日周二 02:36写道：
>>
>>> HI tison, William,
>>>
>>> Thanks for pointing out the problem!
>>>
>>> For the long term, we should support Read-After-Write Consistency. It
>>> seems easy to modify the read-index algorithm to support it: use the index
>>> of the previous request as the read index, instead of the commit index.
>>>
>>> As a work around, we may use the async API with thenApply:
>>> - Client 1: async put k1 as v1 and get future f1
>>> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
>>>
>>> Tsz-Wo
>>>
>>>
>>> On Mon, Aug 28, 2023 at 8:28 AM William Song <sz...@163.com> wrote:
>>>
>>>> Hi tison,
>>>>
>>>> The readIndex provides only the linearizable consistency guarantee. A
>>>> linearizable read returned by RaftServer at least reflects any changes
>>>> already committed and replied to the client. If you require a more strict
>>>> consistency guarantee, like read reflecting the latest write, you may need
>>>> to add extra synchronization logic. I think blocking IO or dependencies on
>>>> completionStage will both work.
>>>>
>>>> Regards,
>>>> William
>>>>
>>>>
>>>> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>>>>
>>>> Hi,
>>>>
>>>> I have asked a related question before[1] about syncing better query
>>>> and applytransaction.
>>>>
>>>> Now, when I dive deeper into the Raft-based system, I notice a new case
>>>> for consistency.
>>>>
>>>> Take the following executions as an example (supposed we implement a
>>>> Map statemachine):
>>>>
>>>> Client 1 put k1 as v1
>>>> Client 1 get k1
>>>>
>>>> Generally, we expect the get requests gets v1. But if these two
>>>> requests are issued in async, it's not always true.
>>>>
>>>> Even the client TCP connection can guarantee that "get k1" goes after
>>>> "put k1", and we do read index for "get k1", the read index returned can be
>>>> lower then "put k1" if it's not yet committed. Then the "get k1" request
>>>> gets an empty value.
>>>>
>>>> I'm not sure what is the best practices to implement read my write for
>>>> a Ratis-based system.
>>>>
>>>> 1. If we're using blocking IO, it won't be an issue cause "get k1" must
>>>> be after "put k1" committed and even applied.
>>>> 2. If we build an extra facade over Ratis like IoTDB does, we can check
>>>> the read conditions as I did at [2].
>>>> 3. But what if we only customize the statemachine and use the Ratis
>>>> client? I don't know which happens before relations I can depend on for
>>>> concurrent query and applytransaction.
>>>>
>>>> Also, in [2] for raft-rs there is one more subtle issue that their
>>>> ReadIndex request can get lost or silently ignored, which causes the caller
>>>> to retry that can even get a much later read index. I'm curious if Ratis
>>>> has the same issue (lose ReadIndex or silently ignore and the caller
>>>> doesn't get response forever).
>>>>
>>>> Best,
>>>> tison.
>>>>
>>>> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
>>>> [2]
>>>> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>>>>
>>>>
>>>>
>
Re: Read my write with async client
Posted by William Song <sz...@163.com>.
> It seems easy to modify the read-index algorithm to support it: use the index of the previous request as the read index, instead of the commit index.
I’ll fire a issue if we plan to implement it.
> We can resolve this issue if we use Ratis as IoTDB that wraps the RaftServer with a facade and handles server-side connection customizedly. That is, the client pipelines requests, but the server sequences the requests and `submitClientRequest` with thenApply chain.
IoTDB’s RaftServer facade is more like a customized client, rather than a server. It pipelines the caller’s requests and then forwards them to the Raft server sequentially.
I think a `Connection` abstract inside which each request casually depends on all the previously requests is what you are asking for. A potential workaround is to implement a `Connection` which accepts async call but use `thenApply` internally to arrange these async calls. It can be officially implemented into Ratis if it’s a common demand.
Hope it helps,
William
> 2023年8月29日 07:14，tison <wa...@gmail.com> 写道：
>
> > use the index of the previous request as the read index, instead of the commit index
>
> Yes. This is exactly what we semantically want to do. Here are two questions we need to answer:
>
> 1. Who drive the new read index process?
> 2. How to fulfill "the previous request's index" as read index?
>
> They can be the same problem.
>
> If the client drives the process, then the only way it knows "the previous request's index" is waiting for the previous request return. This is no better than blocking IO or thenApply.
>
> If the server drives the process, then "the previous request" means the previous request of "this connection". I don't find a connection abstraction in the codebase and this is what I encounter issues on the first place: query/applyTransaction accept requests from all the clients synchronously and not even in order to when they arrive the server.
>
> Best,
> tison.
>
>
> tison <wander4096@gmail.com <ma...@gmail.com>> 于2023年8月29日周二 07:00写道：
>> Thank Wiliam & Tzs Wo!
>>
>> Yeah I just realized that it's somehow a "client-side" or "connection-aspect" issue.
>>
>> The issue of thenApply is the same as blocking IO that we cannot pipeline requests but only send the next read after the pioneer write return from the server.
>>
>> We can resolve this issue if we use Ratis as IoTDB that wraps the RaftServer with a facade and handles server-side connection customizedly. That is, the client pipelines requests, but the server sequences the requests and `submitClientRequest` with thenApply chain. This is like how ZooKeeper's CommitProcessor chains its read/write requests[1].
>>
>> So here we may have another perspective on the question: what are the expected methods of application based on Ratis?
>>
>> If we want users to write applications with RaftClient in App's client-side and only customized StateMachine on App's server side, even use RaftServer as the server facade, then the current query/applyTransaction, and even the fixed one-shot RPC[2] can be a limitation.
>>
>> If we encourage users to wrap RaftServer with their own server facade and use RaftClient in the App's server-side on demand like IoTDB does, then maybe the experience of submitting requests on the server side can be improved; or we provide some examples to do so. (Generally, it should be fine to always use RaftClient in the App's server-side, while it can cause one extra codec, but we already have to send the request to other peers on writes.)
>>
>> Best,
>> tison.
>>
>> [1] https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259
>> [2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm
>>
>> Tsz Wo Sze <szetszwo@gmail.com <ma...@gmail.com>> 于2023年8月29日周二 02:36写道：
>>> HI tison, William,
>>>
>>> Thanks for pointing out the problem!
>>>
>>> For the long term, we should support Read-After-Write Consistency. It seems easy to modify the read-index algorithm to support it: use the index of the previous request as the read index, instead of the commit index.
>>>
>>> As a work around, we may use the async API with thenApply:
>>> - Client 1: async put k1 as v1 and get future f1
>>> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
>>>
>>> Tsz-Wo
>>>
>>>
>>> On Mon, Aug 28, 2023 at 8:28 AM William Song <szywilliam@163.com <ma...@163.com>> wrote:
>>>> Hi tison,
>>>>
>>>> The readIndex provides only the linearizable consistency guarantee. A linearizable read returned by RaftServer at least reflects any changes already committed and replied to the client. If you require a more strict consistency guarantee, like read reflecting the latest write, you may need to add extra synchronization logic. I think blocking IO or dependencies on completionStage will both work.
>>>>
>>>> Regards,
>>>> William
>>>>
>>>>
>>>>> 2023年8月27日 12:20，tison <wander4096@gmail.com <ma...@gmail.com>> 写道：
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have asked a related question before[1] about syncing better query and applytransaction.
>>>>>
>>>>> Now, when I dive deeper into the Raft-based system, I notice a new case for consistency.
>>>>>
>>>>> Take the following executions as an example (supposed we implement a Map statemachine):
>>>>>
>>>>> Client 1 put k1 as v1
>>>>> Client 1 get k1
>>>>>
>>>>> Generally, we expect the get requests gets v1. But if these two requests are issued in async, it's not always true.
>>>>>
>>>>> Even the client TCP connection can guarantee that "get k1" goes after "put k1", and we do read index for "get k1", the read index returned can be lower then "put k1" if it's not yet committed. Then the "get k1" request gets an empty value.
>>>>>
>>>>> I'm not sure what is the best practices to implement read my write for a Ratis-based system.
>>>>>
>>>>> 1. If we're using blocking IO, it won't be an issue cause "get k1" must be after "put k1" committed and even applied.
>>>>> 2. If we build an extra facade over Ratis like IoTDB does, we can check the read conditions as I did at [2].
>>>>> 3. But what if we only customize the statemachine and use the Ratis client? I don't know which happens before relations I can depend on for concurrent query and applytransaction.
>>>>>
>>>>> Also, in [2] for raft-rs there is one more subtle issue that their ReadIndex request can get lost or silently ignored, which causes the caller to retry that can even get a much later read index. I'm curious if Ratis has the same issue (lose ReadIndex or silently ignore and the caller doesn't get response forever).
>>>>>
>>>>> Best,
>>>>> tison.
>>>>>
>>>>> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
>>>>> [2] https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>>>>>
>>>>
Re: Read my write with async client
Posted by tison <wa...@gmail.com>.
> use the index of the previous request as the read index, instead of the
commit index
Yes. This is exactly what we semantically want to do. Here are two
questions we need to answer:
1. Who drive the new read index process?
2. How to fulfill "the previous request's index" as read index?
They can be the same problem.
If the client drives the process, then the only way it knows "the previous
request's index" is waiting for the previous request return. This is no
better than blocking IO or thenApply.
If the server drives the process, then "the previous request" means the
previous request of "this connection". I don't find a connection
abstraction in the codebase and this is what I encounter issues on the
first place: query/applyTransaction accept requests from all the clients
synchronously and not even in order to when they arrive the server.
Best,
tison.
tison <wa...@gmail.com> 于2023年8月29日周二 07:00写道：
> Thank Wiliam & Tzs Wo!
>
> Yeah I just realized that it's somehow a "client-side" or
> "connection-aspect" issue.
>
> The issue of thenApply is the same as blocking IO that we cannot pipeline
> requests but only send the next read after the pioneer write return from
> the server.
>
> We can resolve this issue if we use Ratis as IoTDB that wraps the
> RaftServer with a facade and handles server-side connection customizedly.
> That is, the client pipelines requests, but the server sequences the
> requests and `submitClientRequest` with thenApply chain. This is like how
> ZooKeeper's CommitProcessor chains its read/write requests[1].
>
> So here we may have another perspective on the question: what are the
> expected methods of application based on Ratis?
>
> If we want users to write applications with RaftClient in App's
> client-side and only customized StateMachine on App's server side, even use
> RaftServer as the server facade, then the current query/applyTransaction,
> and even the fixed one-shot RPC[2] can be a limitation.
>
> If we encourage users to wrap RaftServer with their own server facade and
> use RaftClient in the App's server-side on demand like IoTDB does, then
> maybe the experience of submitting requests on the server side can be
> improved; or we provide some examples to do so. (Generally, it should be
> fine to always use RaftClient in the App's server-side, while it can cause
> one extra codec, but we already have to send the request to other peers on
> writes.)
>
> Best,
> tison.
>
> [1]
> https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259
> [2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm
>
> Tsz Wo Sze <sz...@gmail.com> 于2023年8月29日周二 02:36写道：
>
>> HI tison, William,
>>
>> Thanks for pointing out the problem!
>>
>> For the long term, we should support Read-After-Write Consistency. It
>> seems easy to modify the read-index algorithm to support it: use the index
>> of the previous request as the read index, instead of the commit index.
>>
>> As a work around, we may use the async API with thenApply:
>> - Client 1: async put k1 as v1 and get future f1
>> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
>>
>> Tsz-Wo
>>
>>
>> On Mon, Aug 28, 2023 at 8:28 AM William Song <sz...@163.com> wrote:
>>
>>> Hi tison,
>>>
>>> The readIndex provides only the linearizable consistency guarantee. A
>>> linearizable read returned by RaftServer at least reflects any changes
>>> already committed and replied to the client. If you require a more strict
>>> consistency guarantee, like read reflecting the latest write, you may need
>>> to add extra synchronization logic. I think blocking IO or dependencies on
>>> completionStage will both work.
>>>
>>> Regards,
>>> William
>>>
>>>
>>> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>>>
>>> Hi,
>>>
>>> I have asked a related question before[1] about syncing better query
>>> and applytransaction.
>>>
>>> Now, when I dive deeper into the Raft-based system, I notice a new case
>>> for consistency.
>>>
>>> Take the following executions as an example (supposed we implement a Map
>>> statemachine):
>>>
>>> Client 1 put k1 as v1
>>> Client 1 get k1
>>>
>>> Generally, we expect the get requests gets v1. But if these two requests
>>> are issued in async, it's not always true.
>>>
>>> Even the client TCP connection can guarantee that "get k1" goes after
>>> "put k1", and we do read index for "get k1", the read index returned can be
>>> lower then "put k1" if it's not yet committed. Then the "get k1" request
>>> gets an empty value.
>>>
>>> I'm not sure what is the best practices to implement read my write for a
>>> Ratis-based system.
>>>
>>> 1. If we're using blocking IO, it won't be an issue cause "get k1" must
>>> be after "put k1" committed and even applied.
>>> 2. If we build an extra facade over Ratis like IoTDB does, we can check
>>> the read conditions as I did at [2].
>>> 3. But what if we only customize the statemachine and use the Ratis
>>> client? I don't know which happens before relations I can depend on for
>>> concurrent query and applytransaction.
>>>
>>> Also, in [2] for raft-rs there is one more subtle issue that their
>>> ReadIndex request can get lost or silently ignored, which causes the caller
>>> to retry that can even get a much later read index. I'm curious if Ratis
>>> has the same issue (lose ReadIndex or silently ignore and the caller
>>> doesn't get response forever).
>>>
>>> Best,
>>> tison.
>>>
>>> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
>>> [2]
>>> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>>>
>>>
>>>
Re: Read my write with async client
Posted by tison <wa...@gmail.com>.
Thank Wiliam & Tzs Wo!
Yeah I just realized that it's somehow a "client-side" or
"connection-aspect" issue.
The issue of thenApply is the same as blocking IO that we cannot pipeline
requests but only send the next read after the pioneer write return from
the server.
We can resolve this issue if we use Ratis as IoTDB that wraps the
RaftServer with a facade and handles server-side connection customizedly.
That is, the client pipelines requests, but the server sequences the
requests and `submitClientRequest` with thenApply chain. This is like how
ZooKeeper's CommitProcessor chains its read/write requests[1].
So here we may have another perspective on the question: what are the
expected methods of application based on Ratis?
If we want users to write applications with RaftClient in App's client-side
and only customized StateMachine on App's server side, even use RaftServer
as the server facade, then the current query/applyTransaction, and even the
fixed one-shot RPC[2] can be a limitation.
If we encourage users to wrap RaftServer with their own server facade and
use RaftClient in the App's server-side on demand like IoTDB does, then
maybe the experience of submitting requests on the server side can be
improved; or we provide some examples to do so. (Generally, it should be
fine to always use RaftClient in the App's server-side, while it can cause
one extra codec, but we already have to send the request to other peers on
writes.)
Best,
tison.
[1]
https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259
[2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm
Tsz Wo Sze <sz...@gmail.com> 于2023年8月29日周二 02:36写道：
> HI tison, William,
>
> Thanks for pointing out the problem!
>
> For the long term, we should support Read-After-Write Consistency. It
> seems easy to modify the read-index algorithm to support it: use the index
> of the previous request as the read index, instead of the commit index.
>
> As a work around, we may use the async API with thenApply:
> - Client 1: async put k1 as v1 and get future f1
> - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
>
> Tsz-Wo
>
>
> On Mon, Aug 28, 2023 at 8:28 AM William Song <sz...@163.com> wrote:
>
>> Hi tison,
>>
>> The readIndex provides only the linearizable consistency guarantee. A
>> linearizable read returned by RaftServer at least reflects any changes
>> already committed and replied to the client. If you require a more strict
>> consistency guarantee, like read reflecting the latest write, you may need
>> to add extra synchronization logic. I think blocking IO or dependencies on
>> completionStage will both work.
>>
>> Regards,
>> William
>>
>>
>> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>>
>> Hi,
>>
>> I have asked a related question before[1] about syncing better query
>> and applytransaction.
>>
>> Now, when I dive deeper into the Raft-based system, I notice a new case
>> for consistency.
>>
>> Take the following executions as an example (supposed we implement a Map
>> statemachine):
>>
>> Client 1 put k1 as v1
>> Client 1 get k1
>>
>> Generally, we expect the get requests gets v1. But if these two requests
>> are issued in async, it's not always true.
>>
>> Even the client TCP connection can guarantee that "get k1" goes after
>> "put k1", and we do read index for "get k1", the read index returned can be
>> lower then "put k1" if it's not yet committed. Then the "get k1" request
>> gets an empty value.
>>
>> I'm not sure what is the best practices to implement read my write for a
>> Ratis-based system.
>>
>> 1. If we're using blocking IO, it won't be an issue cause "get k1" must
>> be after "put k1" committed and even applied.
>> 2. If we build an extra facade over Ratis like IoTDB does, we can check
>> the read conditions as I did at [2].
>> 3. But what if we only customize the statemachine and use the Ratis
>> client? I don't know which happens before relations I can depend on for
>> concurrent query and applytransaction.
>>
>> Also, in [2] for raft-rs there is one more subtle issue that their
>> ReadIndex request can get lost or silently ignored, which causes the caller
>> to retry that can even get a much later read index. I'm curious if Ratis
>> has the same issue (lose ReadIndex or silently ignore and the caller
>> doesn't get response forever).
>>
>> Best,
>> tison.
>>
>> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
>> [2]
>> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>>
>>
>>
Re: Read my write with async client
Posted by Tsz Wo Sze <sz...@gmail.com>.
HI tison, William,
Thanks for pointing out the problem!
For the long term, we should support Read-After-Write Consistency. It
seems easy to modify the read-index algorithm to support it: use the index
of the previous request as the read index, instead of the commit index.
As a work around, we may use the async API with thenApply:
- Client 1: async put k1 as v1 and get future f1
- Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}).
Tsz-Wo
On Mon, Aug 28, 2023 at 8:28 AM William Song <sz...@163.com> wrote:
> Hi tison,
>
> The readIndex provides only the linearizable consistency guarantee. A
> linearizable read returned by RaftServer at least reflects any changes
> already committed and replied to the client. If you require a more strict
> consistency guarantee, like read reflecting the latest write, you may need
> to add extra synchronization logic. I think blocking IO or dependencies on
> completionStage will both work.
>
> Regards,
> William
>
>
> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>
> Hi,
>
> I have asked a related question before[1] about syncing better query
> and applytransaction.
>
> Now, when I dive deeper into the Raft-based system, I notice a new case
> for consistency.
>
> Take the following executions as an example (supposed we implement a Map
> statemachine):
>
> Client 1 put k1 as v1
> Client 1 get k1
>
> Generally, we expect the get requests gets v1. But if these two requests
> are issued in async, it's not always true.
>
> Even the client TCP connection can guarantee that "get k1" goes after "put
> k1", and we do read index for "get k1", the read index returned can be
> lower then "put k1" if it's not yet committed. Then the "get k1" request
> gets an empty value.
>
> I'm not sure what is the best practices to implement read my write for a
> Ratis-based system.
>
> 1. If we're using blocking IO, it won't be an issue cause "get k1" must be
> after "put k1" committed and even applied.
> 2. If we build an extra facade over Ratis like IoTDB does, we can check
> the read conditions as I did at [2].
> 3. But what if we only customize the statemachine and use the Ratis
> client? I don't know which happens before relations I can depend on for
> concurrent query and applytransaction.
>
> Also, in [2] for raft-rs there is one more subtle issue that their
> ReadIndex request can get lost or silently ignored, which causes the caller
> to retry that can even get a much later read index. I'm curious if Ratis
> has the same issue (lose ReadIndex or silently ignore and the caller
> doesn't get response forever).
>
> Best,
> tison.
>
> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
> [2]
> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>
>
>
Re: Read my write with async client
Posted by William Song <sz...@163.com>.
Hi tison,
The readIndex provides only the linearizable consistency guarantee. A linearizable read returned by RaftServer at least reflects any changes already committed and replied to the client. If you require a more strict consistency guarantee, like read reflecting the latest write, you may need to add extra synchronization logic. I think blocking IO or dependencies on completionStage will both work.
Regards,
William
> 2023年8月27日 12:20，tison <wa...@gmail.com> 写道：
>
> Hi,
>
> I have asked a related question before[1] about syncing better query and applytransaction.
>
> Now, when I dive deeper into the Raft-based system, I notice a new case for consistency.
>
> Take the following executions as an example (supposed we implement a Map statemachine):
>
> Client 1 put k1 as v1
> Client 1 get k1
>
> Generally, we expect the get requests gets v1. But if these two requests are issued in async, it's not always true.
>
> Even the client TCP connection can guarantee that "get k1" goes after "put k1", and we do read index for "get k1", the read index returned can be lower then "put k1" if it's not yet committed. Then the "get k1" request gets an empty value.
>
> I'm not sure what is the best practices to implement read my write for a Ratis-based system.
>
> 1. If we're using blocking IO, it won't be an issue cause "get k1" must be after "put k1" committed and even applied.
> 2. If we build an extra facade over Ratis like IoTDB does, we can check the read conditions as I did at [2].
> 3. But what if we only customize the statemachine and use the Ratis client? I don't know which happens before relations I can depend on for concurrent query and applytransaction.
>
> Also, in [2] for raft-rs there is one more subtle issue that their ReadIndex request can get lost or silently ignored, which causes the caller to retry that can even get a much later read index. I'm curious if Ratis has the same issue (lose ReadIndex or silently ignore and the caller doesn't get response forever).
>
> Best,
> tison.
>
> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s
> [2] https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686
>