You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by Mahesh Garlapati <Ma...@mastercard.com.INVALID> on 2023/12/11 20:54:21 UTC
Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Hello team,
I initiated three nodes, namely n0, n1, and n2, using the following commands:
ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
Based on the logs, it appears that n2 is the leader:
INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
In another scenario:
I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
These errors were resolved by running the append command using the admin client.
I appreciate your assistance in resolving this issues.
Thanks,
Mahesh Reddy
CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Posted by William Song <sz...@163.com>.
Hi Mahesh,
According to the log, I assume you are trying to add a new node n3 into the original group (n0,n1,n2).
In that case, we should provide
OLDPEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002,n3:127.0.0.1:6003.
Since the parameter OLDPEERS represents for the original group while the PEERS represents for the group after append. When `COMPARE_AND_SET` mode is used in `setConfiguration` request,
the leader will first compare between the current configuration and the provided configuration in request (setServersInCurrentConf),
and it will only accept the new configuration (setServersInNewConf) if the current configuration equals to the provided configuration in request.
Please let me know if it works for you.
Best,
William
> 2023年12月13日 02:36，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
> Hello William,
>
> Here is the code & logs https://codefile.io/f/y8UX2pyobP
>
> Please check the logs and let me know why it is failed to add to the configuration.
>
> Thanks
> Mahesh Reddy.
>
> From: William Song <szywilliam@163.com <ma...@163.com>>
> Date: Monday, December 11, 2023 at 8:03 PM
> To: dev@ratis.apache.org <ma...@ratis.apache.org> <dev@ratis.apache.org <ma...@ratis.apache.org>>
> Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
> Hi Mahesh, The apache mailing list allows only plain text and I can't see the attached append. java. Could you please upload to cloud and provide the access link? BTW, I noticed that you are using COMPARE_AND_SET mode in setConfiguration request.
>
>
> Hi Mahesh,
>
>
>
> The apache mailing list allows only plain text and I can't see the attached append.java. Could you please upload to cloud and provide the access link?
>
>
>
> BTW, I noticed that you are using COMPARE_AND_SET mode in setConfiguration request. In that way, it’s required to setServersInCurrentConf with original old group (n0,n1,n2) and setServersInNewConf with the intended new group (n0, n1, n2, n3).
>
>
>
> Thanks,
>
> William
>
>
>
>> 2023年12月12日 11:27，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
>>
>
>> Hello William,
>
>>
>
>> << please find my inline answers >>
>
>>
>
>> Thanks
>
>> Mahesh Reddy.
>
>>
>
>> From: William Song <szywilliam@163.com <ma...@163.com> <ma...@163.com>>
>
>> Date: Monday, December 11, 2023 at 6:35 PM
>
>> To: dev@ratis.apache.org <ma...@ratis.apache.org> <ma...@ratis.apache.org> <dev@ratis.apache.org <ma...@ratis.apache.org>>
>
>> Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
>
>>
>
>> Hi Mahesh, Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;( For membership change examples, please refer to example/membership in latest 3. 0 version. You can find the relevant code
>
>> Hi Mahesh,
>
>>
>
>> Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;(
>
>> << With the help of Tsz, I created append.java(attached) and able to update the Ratis node cluster >>
>
>>
>
>> For membership change examples, please refer to example/membership in latest 3.0 version. You can find the relevant code here https://urldefense.com/v3/__https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$<https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$><https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$><https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3e%3e> <https://urldefense.com/v3/__https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3Chttps://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3E%3Chttps://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3E%3Chttps://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3e%3e%3E>
>
>> <https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3e%3e>
>
>> When changing from (n0, n1, n2) to (n0, n1, n3), you can follow these steps
>
>> 1. Start n3 with an EMPTY group configuration.
>
>> 2. Send a setConfiguration([n0, n1, n3]) request to the existing group.
>
>> 3. Once 2 is executed successfully, you can now safely shutdown n2 and clean up the data.
>
>>
>
>> <<
>
>> In-fact , I tried this approach, but it didn’t worked as excepted
>
>>
>
>> Started 3 nodes n0,n1, n2 with the following commands
>
>> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
>
>> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>> Started n3 node with the following command ID=n3; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers n3:127.0.0.1:6003
>
>> setConfiguration as follows but it is failed (attached append-n3.txt file for logs)
>
>> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
>
>> OLDPEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002,n3:127.0.0.1:6003
>
>> {BIN}/client.sh arithmetic append --peers ${PEERS} --oldPeers ${OLDPEERS}
>
>>
>
>>>>
>
>>
>
>> In arithmetic example, n3 is started with the intended new group(n0, n1, n3) but Step 2 is not executed. Without this, the original group (n0, n1, n2) remains unaware of the new configuration and continues to operate under the assumption that the group consists of (n0, n1, n2). This lead to n3 being rejected and shutdown.
>
>>
>
>> Best Regards,
>
>> William
>
>>
>
>>
>
>>> 2023年12月12日 04:54，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
>>>
>
>>> Hello team,
>
>>>
>
>>> I initiated three nodes, namely n0, n1, and n2, using the following commands:
>
>>>
>
>>> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>>> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>>> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
>>>
>
>>> Based on the logs, it appears that n2 is the leader:
>
>>>
>
>>> INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
>
>>>
>
>>> However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
>
>>>
>
>>> I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
>
>>>
>
>>> In another scenario:
>
>>>
>
>>> I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
>
>>>
>
>>> INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
>
>>>
>
>>> Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
>
>>>
>
>>> 2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
>
>>>
>
>>> These errors were resolved by running the append command using the admin client.
>
>>>
>
>>> I appreciate your assistance in resolving this issues.
>
>>>
>
>>> Thanks,
>
>>> Mahesh Reddy
>
>>>
>
>>> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. <log.txt>
>
>>
>
>> <append-n3.txt>
Re: Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Posted by Mahesh Garlapati <Ma...@mastercard.com.INVALID>.
Hello William,
Here is the code & logs https://codefile.io/f/y8UX2pyobP
Please check the logs and let me know why it is failed to add to the configuration.
Thanks
Mahesh Reddy.
From: William Song <sz...@163.com>
Date: Monday, December 11, 2023 at 8:03 PM
To: dev@ratis.apache.org <de...@ratis.apache.org>
Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Hi Mahesh, The apache mailing list allows only plain text and I can't see the attached append. java. Could you please upload to cloud and provide the access link? BTW, I noticed that you are using COMPARE_AND_SET mode in setConfiguration request.
Hi Mahesh,
The apache mailing list allows only plain text and I can't see the attached append.java. Could you please upload to cloud and provide the access link?
BTW, I noticed that you are using COMPARE_AND_SET mode in setConfiguration request. In that way, it’s required to setServersInCurrentConf with original old group (n0,n1,n2) and setServersInNewConf with the intended new group (n0, n1, n2, n3).
Thanks,
William
> 2023年12月12日 11:27，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
> Hello William,
>
> << please find my inline answers >>
>
> Thanks
> Mahesh Reddy.
>
> From: William Song <szywilliam@163.com <ma...@163.com>>
> Date: Monday, December 11, 2023 at 6:35 PM
> To: dev@ratis.apache.org <ma...@ratis.apache.org> <dev@ratis.apache.org <ma...@ratis.apache.org>>
> Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
>
> Hi Mahesh, Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;( For membership change examples, please refer to example/membership in latest 3. 0 version. You can find the relevant code
> Hi Mahesh,
>
> Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;(
> << With the help of Tsz, I created append.java(attached) and able to update the Ratis node cluster >>
>
> For membership change examples, please refer to example/membership in latest 3.0 version. You can find the relevant code here https://urldefense.com/v3/__https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$<https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$><https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$><https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3e%3e>
><https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$%3e%3e>
> When changing from (n0, n1, n2) to (n0, n1, n3), you can follow these steps
> 1. Start n3 with an EMPTY group configuration.
> 2. Send a setConfiguration([n0, n1, n3]) request to the existing group.
> 3. Once 2 is executed successfully, you can now safely shutdown n2 and clean up the data.
>
> <<
> In-fact , I tried this approach, but it didn’t worked as excepted
>
> Started 3 nodes n0,n1, n2 with the following commands
> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> Started n3 node with the following command ID=n3; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers n3:127.0.0.1:6003
> setConfiguration as follows but it is failed (attached append-n3.txt file for logs)
> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
> OLDPEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002,n3:127.0.0.1:6003
> {BIN}/client.sh arithmetic append --peers ${PEERS} --oldPeers ${OLDPEERS}
>
> >>
>
> In arithmetic example, n3 is started with the intended new group(n0, n1, n3) but Step 2 is not executed. Without this, the original group (n0, n1, n2) remains unaware of the new configuration and continues to operate under the assumption that the group consists of (n0, n1, n2). This lead to n3 being rejected and shutdown.
>
> Best Regards,
> William
>
>
> > 2023年12月12日 04:54，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
> >
> > Hello team,
> >
> > I initiated three nodes, namely n0, n1, and n2, using the following commands:
> >
> > ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> > ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> > ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> >
> > Based on the logs, it appears that n2 is the leader:
> >
> > INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
> >
> > However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
> >
> > I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
> >
> > In another scenario:
> >
> > I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
> >
> > INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
> >
> > Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
> >
> > 2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
> >
> > These errors were resolved by running the append command using the admin client.
> >
> > I appreciate your assistance in resolving this issues.
> >
> > Thanks,
> > Mahesh Reddy
> >
> > CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. <log.txt>
>
> <append-n3.txt>
Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Posted by William Song <sz...@163.com>.
Hi Mahesh,
The apache mailing list allows only plain text and I can't see the attached append.java. Could you please upload to cloud and provide the access link?
BTW, I noticed that you are using COMPARE_AND_SET mode in setConfiguration request. In that way, it’s required to setServersInCurrentConf with original old group (n0,n1,n2) and setServersInNewConf with the intended new group (n0, n1, n2, n3).
Thanks,
William
> 2023年12月12日 11:27，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
> Hello William,
>
> << please find my inline answers >>
>
> Thanks
> Mahesh Reddy.
>
> From: William Song <szywilliam@163.com <ma...@163.com>>
> Date: Monday, December 11, 2023 at 6:35 PM
> To: dev@ratis.apache.org <ma...@ratis.apache.org> <dev@ratis.apache.org <ma...@ratis.apache.org>>
> Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
>
> Hi Mahesh, Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;( For membership change examples, please refer to example/membership in latest 3. 0 version. You can find the relevant code
> Hi Mahesh,
>
> Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;(
> << With the help of Tsz, I created append.java(attached) and able to update the Ratis node cluster >>
>
> For membership change examples, please refer to example/membership in latest 3.0 version. You can find the relevant code here https://urldefense.com/v3/__https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$ <https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$>
>
> When changing from (n0, n1, n2) to (n0, n1, n3), you can follow these steps
> 1. Start n3 with an EMPTY group configuration.
> 2. Send a setConfiguration([n0, n1, n3]) request to the existing group.
> 3. Once 2 is executed successfully, you can now safely shutdown n2 and clean up the data.
>
> <<
> In-fact , I tried this approach, but it didn’t worked as excepted
>
> Started 3 nodes n0,n1, n2 with the following commands
> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> Started n3 node with the following command ID=n3; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers n3:127.0.0.1:6003
> setConfiguration as follows but it is failed (attached append-n3.txt file for logs)
> PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
> OLDPEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002,n3:127.0.0.1:6003
> {BIN}/client.sh arithmetic append --peers ${PEERS} --oldPeers ${OLDPEERS}
>
> >>
>
> In arithmetic example, n3 is started with the intended new group(n0, n1, n3) but Step 2 is not executed. Without this, the original group (n0, n1, n2) remains unaware of the new configuration and continues to operate under the assumption that the group consists of (n0, n1, n2). This lead to n3 being rejected and shutdown.
>
> Best Regards,
> William
>
>
> > 2023年12月12日 04:54，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
> >
> > Hello team,
> >
> > I initiated three nodes, namely n0, n1, and n2, using the following commands:
> >
> > ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> > ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> > ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> >
> > Based on the logs, it appears that n2 is the leader:
> >
> > INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
> >
> > However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
> >
> > I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
> >
> > In another scenario:
> >
> > I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
> >
> > INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
> >
> > Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
> >
> > 2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
> >
> > These errors were resolved by running the append command using the admin client.
> >
> > I appreciate your assistance in resolving this issues.
> >
> > Thanks,
> > Mahesh Reddy
> >
> > CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. <log.txt>
>
> <append-n3.txt>
Re: Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Posted by Mahesh Garlapati <Ma...@mastercard.com.INVALID>.
Hello William,
<< please find my inline answers >>
Thanks
Mahesh Reddy.
From: William Song <sz...@163.com>
Date: Monday, December 11, 2023 at 6:35 PM
To: dev@ratis.apache.org <de...@ratis.apache.org>
Subject: {EXTERNAL} Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Hi Mahesh, Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;( For membership change examples, please refer to example/membership in latest 3. 0 version. You can find the relevant code
Hi Mahesh,
Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;(
<< With the help of Tsz, I created append.java(attached) and able to update the Ratis node cluster >>
For membership change examples, please refer to example/membership in latest 3.0 version. You can find the relevant code here https://urldefense.com/v3/__https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$<https://urldefense.com/v3/__https:/github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java*L68__;Iw!!NDdRaFrjhKsg!r-vVzYJp-wYyEcDzkCI8HX2lf84hLyO_WLw18MT2GtsI4AIxn1_ERJXhwurxqnJ_qqYfAfUo6g83HTToeRwjV7406O0$>
When changing from (n0, n1, n2) to (n0, n1, n3), you can follow these steps
1. Start n3 with an EMPTY group configuration.
2. Send a setConfiguration([n0, n1, n3]) request to the existing group.
3. Once 2 is executed successfully, you can now safely shutdown n2 and clean up the data.
<<
In-fact , I tried this approach, but it didn’t worked as excepted
Started 3 nodes n0,n1, n2 with the following commands
PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
Started n3 node with the following command ID=n3; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers n3:127.0.0.1:6003
setConfiguration as follows but it is failed (attached append-n3.txt file for logs)
PEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
OLDPEERS=n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002,n3:127.0.0.1:6003
{BIN}/client.sh arithmetic append --peers ${PEERS} --oldPeers ${OLDPEERS}
>>
In arithmetic example, n3 is started with the intended new group(n0, n1, n3) but Step 2 is not executed. Without this, the original group (n0, n1, n2) remains unaware of the new configuration and continues to operate under the assumption that the group consists of (n0, n1, n2). This lead to n3 being rejected and shutdown.
Best Regards,
William
> 2023年12月12日 04:54，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
> Hello team,
>
> I initiated three nodes, namely n0, n1, and n2, using the following commands:
>
> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
> Based on the logs, it appears that n2 is the leader:
>
> INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
>
> However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
>
> I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
>
> In another scenario:
>
> I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
>
> INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
>
> Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
>
> 2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
>
> These errors were resolved by running the append command using the admin client.
>
> I appreciate your assistance in resolving this issues.
>
> Thanks,
> Mahesh Reddy
>
> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. <log.txt>
Re: Assistance Needed with Node Initialization and Role Changes in Ratis Cluster
Posted by William Song <sz...@163.com>.
Hi Mahesh,
Thanks for exploring Ratis! Unfortunately the arithmetic example does not support role changes currently ;(
For membership change examples, please refer to example/membership in latest 3.0 version. You can find the relevant code here https://github.com/apache/ratis/blob/53831534c69309688ce379006363e645bf42b654/ratis-examples/src/main/java/org/apache/ratis/examples/membership/server/RaftCluster.java#L68
When changing from (n0, n1, n2) to (n0, n1, n3), you can follow these steps
1. Start n3 with an EMPTY group configuration.
2. Send a setConfiguration([n0, n1, n3]) request to the existing group.
3. Once 2 is executed successfully, you can now safely shutdown n2 and clean up the data.
In arithmetic example, n3 is started with the intended new group(n0, n1, n3) but Step 2 is not executed. Without this, the original group (n0, n1, n2) remains unaware of the new configuration and continues to operate under the assumption that the group consists of (n0, n1, n2). This lead to n3 being rejected and shutdown.
Best Regards,
William
> 2023年12月12日 04:54，Mahesh Garlapati <Ma...@mastercard.com.INVALID> 写道：
>
> Hello team,
>
> I initiated three nodes, namely n0, n1, and n2, using the following commands:
>
> ID=n0; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n1; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
> ID=n2; ${BIN}/server.sh arithmetic server --id ${ID} --storage /tmp/ratis/${ID} --peers ${PEERS}
>
> Based on the logs, it appears that n2 is the leader:
>
> INFO StateMachine:166 - LEADER:n2-1: a = 10 = 10
>
> However, when I attempted to stop the n2 node (leader) and start a new node, n3, with peers n0, n1, and n3, the n3 server failed to start. Attached are the log files for review.
>
> I am seeking assistance in understanding the reason for this error preventing the successful start of the n3 node.
>
> In another scenario:
>
> I started three nodes, n0, n1, and n2, using similar commands, and logs indicated that n1 was the leader:
>
> INFO StateMachine:166 - LEADER:n1-1: a = 10 = 10
>
> Upon stopping the non-leader node, n2, I attempted to start a new node, n3, with peers n0, n1, and n3. The n3 server started but generated errors such as:
>
> 2023-12-11 12:43:13 INFO RaftServer$Division:377 - n3@group-6F7570313233: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
>
> These errors were resolved by running the append command using the admin client.
>
> I appreciate your assistance in resolving this issues.
>
> Thanks,
> Mahesh Reddy
>
> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. <log.txt>