[GitHub] [iotdb] jixuan1989 commented on issue #3954: Integrate Apache Ratis to help manage Raft status

[GitBox <gi...@apache.org>]

jixuan1989 commented on issue #3954:
URL: https://github.com/apache/iotdb/issues/3954#issuecomment-920919425


   Hi folks,
   
   Very glad to see the deep discussion about this topic.
   
   Actually we had several similar discussions in the last 2 years. Even last year, I did not give up trying to use Ratis [1]. However, we finally give up the solution.
   
   Some guys may read the paper "Apache IoTDB 分布式架构设计" [2]. The paper is written in 2019, and at that time, we used sofa-jraft[3], and we should release the cluster version at that time but finally, we gave up. The experience told us using an existing implementation is not as beautiful as we thought. 
   In that version, I asked many times "why you design classes and codes like this", the answer is always "I have to because of the underlying implementation". 
    
   We finally blame the performance problem and architecture scalability problem on the third-part library. I know the conclusion may be not correct, but it is hard to guarantee whether the history will replay again...
   
   What is more important, Raft and multi-raft are not our destinations. We believe in the time series management scenario, Raft is over-heavy. Using an existing implementation will limit us finally.  and I do not want to re-implement the cluster module once again at that time...
   
   I know our implementation may have many bugs and we omit many corner cases now. But it may not be a bad thing. The progress will let us understand the system deeply. 
   
   IMO, we really need to:
   1. if we can decouple the implementation, it is better. (But it seems there is few things to do according to Tian Jiang's above opinion)
   2. If we consider others have been fully tested, we can learn their test cases to make our implementation more robost.
   
   > The answer must be YES, but WHEN? One year or two?
   
   When writing here, I remember another discussion: when we want to implement the monitoring module for IoTDB, which 3rd library should we use?  micrometer, dropwizard or others? Some of them have rich features but some of them have better performance. So, how to choose? Finally, we decide to extract a framework and let users choose what they want to use.
   
   So, if we can do some refactor and then allow let users to switch the implementation, I can accept that.  
   
   But it is hard to convince me to give up our implementation. In my view, it is a kind of drinking the poison to kill the thirsty.
   
   
   [1] https://lists.apache.org/thread.html/3c49bc234415417511226c7032009868f9d3475026adf07e2e41f99f%40%3Cdev.ratis.apache.org%3E
   [2] http://scis.scichina.com/cn/2020/SSI-2019-0189.pdf
   [3] https://github.com/sofastack/sofa-jraft


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org