You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by William Song <sz...@163.com> on 2022/09/07 15:38:45 UTC
Race condition in LeaderStateImpl
Hi,
We found abnormal behaviors of GrpcLogAppender in a recent run, please refer to https://issues.apache.org/jira/browse/RATIS-1699 <https://issues.apache.org/jira/browse/RATIS-1699> for detailed error log and problem description.
We think that this problem roots in the data race in LeaderStateImpl. Its method stop() and restart(LogAppender) can be called concurrently, and under certain event sequence, will cause this weird situation. Maybe we can add synchronizations to coordinate these two methods. The event sequence and full logs are also provided in this issue.
Please help me to confirm whether our analysis stands. Thanks in advance!
Regards,
William
Re: Race condition in LeaderStateImpl
Posted by William Song <sz...@163.com>.
Hi Yaolong,
Thanks for letting me know the current status. Hoping everything works out fine for 2.4.0 :)
Regards,
William
> 2022年9月8日 12:59，Yaolong Liu <ly...@163.com> 写道：
>
> Hi William,
>
>
> Since Ratis 2.4.0 plans to introduce RATIS-1677 and RATIS-1694, which is currently in the test phase, so the release progress is temporarily blocked.
> see https://github.com/apache/ratis/pull/718
>
>
> -Yaolong
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2022-09-08 10:28:04, "William Song" <sz...@163.com> wrote:
>> Hi Tsz-wo,
>>
>> Thanks for confirming this! Hoping this fix can catch 2.4.0 release.
>>
>> BTW, I see that the community has closed Release 2.4.0-rc1 voting process due to compatibility issues. What is the current progress? Our community (IoTDB) plans to release a major version 0.14.0, which relies on features that will introduced in 2.4.0. Please let me know if there’s anything we can help.
>>
>> Regards,
>> William
>>
>>> 2022年9月8日 08:31，Tsz Wo Sze <sz...@gmail.com> 写道：
>>>
>>> Hi William,
>>>
>>> Thanks a lot for reporting the problem! Fortunately, the race
>>> condition happens during shutdown since "SegmentRaftLog is closed" as
>>> mentioned in the description of RATIS-1699. Of course, we would like
>>> to fix it. I will check it for more details.
>>>
>>> Tsz-Wo
>>>
>>> On Wed, Sep 7, 2022 at 11:39 PM William Song <sz...@163.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We found abnormal behaviors of GrpcLogAppender in a recent run, please refer to https://issues.apache.org/jira/browse/RATIS-1699 <https://issues.apache.org/jira/browse/RATIS-1699> for detailed error log and problem description.
>>>>
>>>> We think that this problem roots in the data race in LeaderStateImpl. Its method stop() and restart(LogAppender) can be called concurrently, and under certain event sequence, will cause this weird situation. Maybe we can add synchronizations to coordinate these two methods. The event sequence and full logs are also provided in this issue.
>>>>
>>>> Please help me to confirm whether our analysis stands. Thanks in advance!
>>>>
>>>> Regards,
>>>> William
Re:Re: Race condition in LeaderStateImpl
Posted by Yaolong Liu <ly...@163.com>.
Hi William,
Since Ratis 2.4.0 plans to introduce RATIS-1677 and RATIS-1694, which is currently in the test phase, so the release progress is temporarily blocked.
see https://github.com/apache/ratis/pull/718
-Yaolong
At 2022-09-08 10:28:04, "William Song" <sz...@163.com> wrote:
>Hi Tsz-wo,
>
>Thanks for confirming this! Hoping this fix can catch 2.4.0 release.
>
>BTW, I see that the community has closed Release 2.4.0-rc1 voting process due to compatibility issues. What is the current progress? Our community (IoTDB) plans to release a major version 0.14.0, which relies on features that will introduced in 2.4.0. Please let me know if there’s anything we can help.
>
>Regards,
>William
>
>> 2022年9月8日 08:31，Tsz Wo Sze <sz...@gmail.com> 写道：
>>
>> Hi William,
>>
>> Thanks a lot for reporting the problem! Fortunately, the race
>> condition happens during shutdown since "SegmentRaftLog is closed" as
>> mentioned in the description of RATIS-1699. Of course, we would like
>> to fix it. I will check it for more details.
>>
>> Tsz-Wo
>>
>> On Wed, Sep 7, 2022 at 11:39 PM William Song <sz...@163.com> wrote:
>>>
>>> Hi,
>>>
>>> We found abnormal behaviors of GrpcLogAppender in a recent run, please refer to https://issues.apache.org/jira/browse/RATIS-1699 <https://issues.apache.org/jira/browse/RATIS-1699> for detailed error log and problem description.
>>>
>>> We think that this problem roots in the data race in LeaderStateImpl. Its method stop() and restart(LogAppender) can be called concurrently, and under certain event sequence, will cause this weird situation. Maybe we can add synchronizations to coordinate these two methods. The event sequence and full logs are also provided in this issue.
>>>
>>> Please help me to confirm whether our analysis stands. Thanks in advance!
>>>
>>> Regards,
>>> William
Re: Race condition in LeaderStateImpl
Posted by William Song <sz...@163.com>.
Hi Tsz-wo,
Thanks for confirming this! Hoping this fix can catch 2.4.0 release.
BTW, I see that the community has closed Release 2.4.0-rc1 voting process due to compatibility issues. What is the current progress? Our community (IoTDB) plans to release a major version 0.14.0, which relies on features that will introduced in 2.4.0. Please let me know if there’s anything we can help.
Regards,
William
> 2022年9月8日 08:31，Tsz Wo Sze <sz...@gmail.com> 写道：
>
> Hi William,
>
> Thanks a lot for reporting the problem! Fortunately, the race
> condition happens during shutdown since "SegmentRaftLog is closed" as
> mentioned in the description of RATIS-1699. Of course, we would like
> to fix it. I will check it for more details.
>
> Tsz-Wo
>
> On Wed, Sep 7, 2022 at 11:39 PM William Song <sz...@163.com> wrote:
>>
>> Hi,
>>
>> We found abnormal behaviors of GrpcLogAppender in a recent run, please refer to https://issues.apache.org/jira/browse/RATIS-1699 <https://issues.apache.org/jira/browse/RATIS-1699> for detailed error log and problem description.
>>
>> We think that this problem roots in the data race in LeaderStateImpl. Its method stop() and restart(LogAppender) can be called concurrently, and under certain event sequence, will cause this weird situation. Maybe we can add synchronizations to coordinate these two methods. The event sequence and full logs are also provided in this issue.
>>
>> Please help me to confirm whether our analysis stands. Thanks in advance!
>>
>> Regards,
>> William
Re: Race condition in LeaderStateImpl
Posted by Tsz Wo Sze <sz...@gmail.com>.
Hi William,
Thanks a lot for reporting the problem! Fortunately, the race
condition happens during shutdown since "SegmentRaftLog is closed" as
mentioned in the description of RATIS-1699. Of course, we would like
to fix it. I will check it for more details.
Tsz-Wo
On Wed, Sep 7, 2022 at 11:39 PM William Song <sz...@163.com> wrote:
>
> Hi,
>
> We found abnormal behaviors of GrpcLogAppender in a recent run, please refer to https://issues.apache.org/jira/browse/RATIS-1699 <https://issues.apache.org/jira/browse/RATIS-1699> for detailed error log and problem description.
>
> We think that this problem roots in the data race in LeaderStateImpl. Its method stop() and restart(LogAppender) can be called concurrently, and under certain event sequence, will cause this weird situation. Maybe we can add synchronizations to coordinate these two methods. The event sequence and full logs are also provided in this issue.
>
> Please help me to confirm whether our analysis stands. Thanks in advance!
>
> Regards,
> William