You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by William Song <sz...@163.com> on 2023/07/19 02:13:52 UTC

[PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus

Hi dev,

I'd like to draw your attention to an existing issue in our current read consistency level within the RatisConsensus module. As it stands, the default level is set to "query statemachine directly”, which, while latency-friendly, has led to user-reported bugs. Specifically, these bugs relate to the production of inconsistent results in subsequent SQL queries during a restart, creating a phantom read problem that may be confusing for our users.

To address this issue, I propose that we temporarily increase the read consistency level to linearizable read during restarts. This will ensure that we maintain data consistency during the critical recovery period. Once the cluster has successfully finished recovering from previous logs, we can then revert to the default consistency level.

You can find more details about this proposed solution in the linked pull request: https://github.com/apache/iotdb/pull/10597。

**Please note** that this change may affect module (including CQ, schema region, and data region) that calls RatisConsensus.read during the restart process. In such cases, a RatisUnderRecoveryException may be returned, indicating that RatisConsensus cannot serve read requests while it's replaying RaftLog. Therefore, we strongly encourage the affected modules to handle this situation appropriately, such as implementing a retry mechanism.

I look forward to hearing your thoughts on this proposal. Your feedback and suggestions will be appreciated.

Regards
William Song

Re: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus

Posted by William Song <sz...@163.com>.
Hi Chris,

> Trust lost easily, and hard to regain.

Can’t agree more. 

Maybe we shall consider implementing lease read to achieve consistency & latency balance later after pull/10597 <https://github.com/apache/iotdb/pull/10597>. CC Xinyu.

William

> 2023年7月19日 14:46，Christofer Dutz <ch...@c-ware.de> 写道：
> 
> Hi,
> 
> I agree that it’s better to have the defaults produce safer (more consistent) results and document optimization options for users, that want/need them and know about potential drawbacks.
> Admittedly I’m not yet too deep in the internals of IoTDB, but at least this would be my expectation on a user-level.
> 
> I’m currently reviewing our “competitor” solutions and inconsistencies were what made me dislike the one or the other solution instantly. Trust lost easily, and hard to regain.
> 
> Chris
> 
> 
> Von: William Song <sz...@163.com>
> Datum: Mittwoch, 19. Juli 2023 um 04:14
> An: dev@iotdb.apache.org <de...@iotdb.apache.org>
> Betreff: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus
> Hi dev,
> 
> I'd like to draw your attention to an existing issue in our current read consistency level within the RatisConsensus module. As it stands, the default level is set to "query statemachine directly”, which, while latency-friendly, has led to user-reported bugs. Specifically, these bugs relate to the production of inconsistent results in subsequent SQL queries during a restart, creating a phantom read problem that may be confusing for our users.
> 
> To address this issue, I propose that we temporarily increase the read consistency level to linearizable read during restarts. This will ensure that we maintain data consistency during the critical recovery period. Once the cluster has successfully finished recovering from previous logs, we can then revert to the default consistency level.
> 
> You can find more details about this proposed solution in the linked pull request: https://github.com/apache/iotdb/pull/10597。
> 
> **Please note** that this change may affect module (including CQ, schema region, and data region) that calls RatisConsensus.read during the restart process. In such cases, a RatisUnderRecoveryException may be returned, indicating that RatisConsensus cannot serve read requests while it's replaying RaftLog. Therefore, we strongly encourage the affected modules to handle this situation appropriately, such as implementing a retry mechanism.
> 
> I look forward to hearing your thoughts on this proposal. Your feedback and suggestions will be appreciated.
> 
> Regards
> William Song

AW: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,

I agree that it’s better to have the defaults produce safer (more consistent) results and document optimization options for users, that want/need them and know about potential drawbacks.
Admittedly I’m not yet too deep in the internals of IoTDB, but at least this would be my expectation on a user-level.

I’m currently reviewing our “competitor” solutions and inconsistencies were what made me dislike the one or the other solution instantly. Trust lost easily, and hard to regain.

Chris


Von: William Song <sz...@163.com>
Datum: Mittwoch, 19. Juli 2023 um 04:14
An: dev@iotdb.apache.org <de...@iotdb.apache.org>
Betreff: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus
Hi dev,

I'd like to draw your attention to an existing issue in our current read consistency level within the RatisConsensus module. As it stands, the default level is set to "query statemachine directly”, which, while latency-friendly, has led to user-reported bugs. Specifically, these bugs relate to the production of inconsistent results in subsequent SQL queries during a restart, creating a phantom read problem that may be confusing for our users.

To address this issue, I propose that we temporarily increase the read consistency level to linearizable read during restarts. This will ensure that we maintain data consistency during the critical recovery period. Once the cluster has successfully finished recovering from previous logs, we can then revert to the default consistency level.

You can find more details about this proposed solution in the linked pull request: https://github.com/apache/iotdb/pull/10597。

**Please note** that this change may affect module (including CQ, schema region, and data region) that calls RatisConsensus.read during the restart process. In such cases, a RatisUnderRecoveryException may be returned, indicating that RatisConsensus cannot serve read requests while it's replaying RaftLog. Therefore, we strongly encourage the affected modules to handle this situation appropriately, such as implementing a retry mechanism.

I look forward to hearing your thoughts on this proposal. Your feedback and suggestions will be appreciated.

Regards
William Song